Packaging pre-built binaries with nix

Here’s the scenario: You have a nix environment all set up with all the dependencies you need for working on your next awesome project. All but one. nixpkgs doesn’t have the version you want. Fortunately, there’s a static binary file on their GitHub page. So should you just manually download it every time you set your project up, or should you write a Nix package that builds it from source?

None of em! I don’t want to maintain any bash scripts to do that. I just want to load up the Nix environment, and start.

That was pretty much what I ran into today, I wanted to have postgrest in my Nix environment so I hopped on nixpkgs search, only to find that the existing versions are quite old, as well as having to build the project from source. Which I didn’t want to do due to my limited LTE bandwidth. Why should I? postgrest has pre-built, static binaries in its GitHub releases page. Is there any way I can make use of that instead? And that’s what I set off to do today.

tl;dr: Binaries are built, me copy. Me save mobile data. Happy.

Disclaimer: I’m a Nix newbie.

Setup

I need to import the postgrest flake to an existing flake of another project. The project’s directory has a nix folder for all the packages that don’t exist in nixpkgs, like so:

nix
└── postgrest
    ├── flake.lock
    └── flake.nix

The flake.lock gets automatically generated by Nix when it finds out that it doesn’t exist yet when running nix build; no need to create that. Here’s the starting flake.nix:

{
  description = "REST API for any Postgres database";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
  };

  outputs = {self, nixpkgs}: { };
}

A basic flake file has a set of inputs and outputs. The output here will be the postgrest package.

Time to throw the binary in Nix.

I have no idea how to build things in flakes, so consulting the wiki is pretty much a requirement ¹! It covers things like how to enable flakes, and I won’t bother covering because I’ll only make a worse version of it.

It tells me about both the inputs and outputs schema. But because I only need nixpkgs for this one, there’s not much else for me to add to the inputs.

Here’s the output schema:

{ self, ... }@inputs:
{
  # Executed by `nix flake check`
  checks."<system>"."<name>" = derivation;
  # Executed by `nix build .#<name>`
  packages."<system>"."<name>" = derivation;
  # Executed by `nix build .`
  defaultPackage."<system>" = derivation;
  # Executed by `nix run .#<name>`
  apps."<system>"."<name>" = {
    type = "app";
    program = "<store-path>";
  };
  # Executed by `nix run . -- <args?>`
  defaultApp."<system>" = { type = "app"; program = "..."; };

  # Used for nixpkgs packages, also accessible via `nix build .#<name>`
  legacyPackages."<system>"."<name>" = derivation;
  # Default overlay, consumed by other flakes
  overlay = final: prev: { };
  # Same idea as overlay but a list or attrset of them.
  overlays = {};
  # Default module, consumed by other flakes
  nixosModule = { config }: { options = {}; config = {}; };
  # Same idea as nixosModule but a list or attrset of them.
  nixosModules = {};
  # Used with `nixos-rebuild --flake .#<hostname>`
  # nixosConfigurations."<hostname>".config.system.build.toplevel must be a derivation
  nixosConfigurations."<hostname>" = {};
  # Used by `nix develop`
  devShell."<system>" = derivation;
  # Used by `nix develop .#<name>`
  devShells."<system>"."<name>" = derivation;
  # Hydra build jobs
  hydraJobs."<attr>"."<system>" = derivation;
  # Used by `nix flake init -t <flake>`
  defaultTemplate = {
    path = "<store-path>";
    description = "template description goes here?";
  };
  # Used by `nix flake init -t <flake>#<name>`
  templates."<name>" = { path = "<store-path>"; description = ""; };
}

That’s a lot.

outputs is a lambda with a set as its argument. Since nix functions can only have one argument, putting the stuff you need in a set is how you get around that restriction. {self, ...} is some form of pattern matching the fields in a set, and @inputs binds the set to input. Cool. The latter isn’t that useful to me in this scenario though, so I’ll omit that. It seems that in every flake outputs, self must be there. I don’t understand what self is, but I’ll leave that for another time.

With the schema, there are two fields that seem important: packages, and defaultPackage. packages would be useful if I wanted to have multiple versions of postgrest available for me to use like postgres-8-0-0 and postgrest-9-0-0, but I don’t! I only need the latest version, which is 9.0.0 at the time of writing. So we can ignore that, and I’ll use defaultPackage instead.

Here’s what we have so far:

{
  description = "REST API for any Postgres database";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
  };

  outputs = {self, nixpkgs}: {
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };

      stdenv.mkDerivation rec {
        name = "postgrest-${version}";

        version = "9.0.0";

        # I still lack stuff here!
      };
  };
}

with import nixpkgs {system = "x86_64-linux"}; spares me from having to qualify everything like nixpkgs.system."x86_64".stdenv.mkDerivation which is handy ². This brings stdenv into scope, and has mkDerivation which, from the name, makes a derivation; something I need for defaultPackage.<system>. Unfortunately, I couldn’t find any official documentation for mkDerivation that specifies every single field usable in it. Maybe it exists and that I just suck at Googling. That is definitely possible. There are some examples ³, especially in the wild.

rec allows me to refer to the set’s own fields within it. I’m using the field version and interpolated it in name!

Alright that’s it for the setup. Time to fetch the binary.

Fetching the binary

The wiki has an example for fetching stuff from a URL, and using it in mkDerivation ⁴.

src = fetchurl {
  url = "https://download.studio.link/releases/v${version}-stable/linux/studio-link-standalone-v${version}.tar.gz";
  sha256 = "sha256-4CkijAlenhht8tyk3nBULaBPE0GBf6DVII699/RmmWI=";
};

So it looks like I need two things, a url which can be a tar file, and a sha256. The sha256 field is used to make something impure a little bit less unpredictable. If the release were somehow to change under the same name, then it would fail cause the SHA would have a different signature.

But… how does one get the SHA? A trick is to just leave it blank. Nix will inform you and make a comparison of the expected vs actual signature.

Add this in the outputs schema:

src = pkgs.fetchurl {
  # Remember `rec`!
  url = "https://github.com/PostgREST/postgrest/releases/download/v${version}/postgrest-v${version}-linux-static-x64.tar.xz";
  sha256 = "";
};

and run nix build in the directory with this flake.nix file.

sekun@nixos ~/P/g/n/postgrest (feature/postgrest)> nix build
warning: Git tree '/home/sekun/Projects/gnawex' is dirty
warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
error: hash mismatch in fixed-output derivation '/nix/store/mag8ly8f0rlw5dqxj7ir8maa1bqgkyxv-postgrest-v9.0.0-linux-static-x64.tar.xz.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-6kgh6heVV7qNcNzcXTiqbVyhfsSV9u5/S3skto6Uzz4=
error: 1 dependencies of derivation '/nix/store/yg6adsask3s2sg636m5dwy0c79dadg9g-postgrest-9.0.0.drv' failed to build
sekun@nixos ~/P/g/n/postgrest (feature/postgrest) [1]>

It does fail, as expected. This is how it’ll be if ever the signature were to change. But there it is! It tells us what we specified, and what Nix got. So let’s yoink that and slap it in the schema.

src = pkgs.fetchurl {
  url = "https://github.com/PostgREST/postgrest/releases/download/v${version}/postgrest-v${version}-linux-static-x64.tar.xz";
  sha256 = "sha256-6kgh6heVV7qNcNzcXTiqbVyhfsSV9u5/S3skto6Uzz4=";
};

Now that we have the binary, all that’s left is to install it.

Installing

Going back to the example in ⁴:

stdenv.mkDerivation rec {
  name = "studio-link-${version}";

  version = "21.07.0";

  src = fetchurl {
    url = "https://download.studio.link/releases/v${version}-stable/linux/studio-link-standalone-v${version}.tar.gz";
    sha256 = "sha256-4CkijAlenhht8tyk3nBULaBPE0GBf6DVII699/RmmWI=";
  };

  nativeBuildInputs = [
    autoPatchelfHook
  ];

  buildInputs = [
    alsaLib
    openssl
    zlib
    pulseaudio
  ];

  sourceRoot = ".";

  installPhase = ''
    install -m755 -D studio-link-standalone-v${version} $out/bin/studio-link
  '';

  meta = with lib; {
    homepage = "https://studio-link.com";
    description = "Voip transfer";
    platforms = platforms.linux;
  };
}

We can ignore nativeBuildInputs and buildInputs since those are used for declaring what dependencies should be there when building something. In this case, there’s nothing to build because we just got a pre-built binary. Nor do we have to provide any runtime dependencies because it’s a static binary. That leaves sourceRoot, installPhase, and meta left. I tried looking for more information about sourceRoot found some explanations but I was left unsure if I needed it. Let’s leave that out for now. We do need installPhase since we have to send it off to the Nix store. It looks like I can reuse this without much changes.

installPhase = ''
install -m755 -D postgrest $out/bin/postgrest
'';

I have no idea what install is. And as usual, check the manual/wiki! The manual description tells me it’s how one copies files while setting attributes. -m755 sets the permissions to 755, postgrest is the source, and $out/bin/postgrest is the target. $out is set by Nix, which points to the Nix store with the package’s name for the folder. Alright, cool!

Time to run nix build again to see if this works.

…and it doesn’t.

sekun@nixos ~/P/g/n/postgrest (feature/postgrest) [1]> nix build
warning: Git tree '/home/sekun/Projects/gnawex' is dirty
error: builder for '/nix/store/jsxk8q3handkprh5ma102v8y1dig9k77-postgrest-9.0.0.drv' failed with exit code 1;
       last 3 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/644yqp1y3cgw45qfqsbxb013hm4r2zw6-postgrest-v9.0.0-linux-static-x64.tar.xz
       > unpacker appears to have produced no directories
       For full logs, run 'nix log /nix/store/jsxk8q3handkprh5ma102v8y1dig9k77-postgrest-9.0.0.drv'.

The error seems to point out that it couldn’t unpack the tar anywhere. Running nix log /nix/store/jsxk8q3handkprh5ma102v8y1dig9k77-postgrest-9.0.0.drv seems to tell me the same thing.

Alright alright, let’s take a look at what the manual has to say about sourceRoot:

After running unpackPhase, the generic builder changes the current directory to the directory created by unpacking the sources. If there are multiple source directories, you should set sourceRoot to the name of the intended directory. Set sourceRoot = “.”; if you use srcs and control the unpack phase yourself.

This isn’t really so helpful because I’m not using srcs, nor am I using unpackPhase, nor am I unpacking multiple sources! I am however specifying a remote file as src with fetchurl, which does seem to unpack it. I have no clue what fetchurl does because the manual doesn’t seem to cover it ⁵. No idea what else to do here so I’ll just follow the suggestion of adding sourceRoot ".". Run nix build again, and see it finally work!

Here’s the final flake:

{
  description = "REST API for any Postgres database";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
  };

  outputs = {self, nixpkgs}: {
    defaultPackage.x86_64-linux =
      with import nixpkgs { system = "x86_64-linux"; };

      stdenv.mkDerivation rec {
        name = "postgrest-${version}";

        version = "9.0.0";

        # https://nixos.wiki/wiki/Packaging/Binaries
        src = pkgs.fetchurl {
          url = "https://github.com/PostgREST/postgrest/releases/download/v${version}/postgrest-v${version}-linux-static-x64.tar.xz";
          sha256 = "sha256-6kgh6heVV7qNcNzcXTiqbVyhfsSV9u5/S3skto6Uzz4=";
        };

        sourceRoot = ".";

        installPhase = ''
        install -m755 -D postgrest $out/bin/postgrest
        '';

        meta = with lib; {
          homepage = "https://postgrest.org";
          description = "REST API for any Postgres database";
          platforms = platforms.linux;
        };
      };
  };
}

Importing a local flake to another flake

Here’s the flake file that needs postgrest:

{
  description = "An independent MouseHunt marketplace";

  inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs";
     masterpkgs.url = "github:NixOS/nixpkgs/master";
     flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, masterpkgs, flake-utils }:
    flake-utils.lib.eachSystem [ "x86_64-linux" ] (system:
      let pkgs = nixpkgs.legacyPackages.${system};
          postgrest = postgrestPkg.defaultPackage.${system};
          lib =  nixpkgs.lib;

      in {
        devShell = pkgs.mkShell rec {
          buildInputs = [
            masterpkgs.legacyPackages.${system}.pgadmin4
          ];
        };
      });
}

Oh, you don’t know what GNAWEX ⁶ is? Well, it’s just an app I’ve been working on for a video game I’ve been playing for nearly 12 years. I’m not addicted, I swear. I’m only doing this to learn PostgreSQL’s cool features!

Besides using flake-utils to make handling different <system>s more convenient, the flake does look pretty much the same. Now how does one refer to this local postgrest flake in this flake? Fortunately, the wiki ¹ has an example in the inputs schema section:

# local directories (for absolute paths you can omit 'path:')
directory-example.url = "path:/path/to/repo";

We need to keep path: since we need a relative path since the postgrest flake is in ./nix/postgrest/flake.nix. Add this to the flake file that needs it, in its inputs set:

inputs = {
  # ...
  postgrestPkg.url = "path:./nix/postgrest";
};

postgrestPkg can be anything, but I’m naming it postgrestPkg to avoid confusion with the actual postgrest package. Then, for convenience, I added this in the let expression:

postgrest = postgrestPkg.defaultPackage.${system};

This binds postgres-9-0-0 to postgrest, which I use in devShell’s buildInputs. Here’s the final flake:

{
  description = "An independent MouseHunt marketplace";

  inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs";
     masterpkgs.url = "github:NixOS/nixpkgs/master";
     postgrestPkg.url = "path:./nix/postgrest"; # New!
     flake-utils.url = "github:numtide/flake-utils";
  };

                                         # V Add this one. Order matters.
  outputs = { self, nixpkgs, masterpkgs, postgrestPkg, flake-utils }:
    flake-utils.lib.eachSystem [ "x86_64-linux" ] (system:
      let pkgs = nixpkgs.legacyPackages.${system};
          postgrest = postgrestPkg.defaultPackage.${system}; # For convenience
          lib =  nixpkgs.lib;

      in {
        devShell = pkgs.mkShell rec {
          buildInputs = [
            postgrest # A shiny `postgres` package!
            masterpkgs.legacyPackages.${system}.pgadmin4
          ];
        };
      });
}

Now postgrest is available in the shell environment:

direnv: loading ~/Projects/gnawex/.envrc
direnv: using flake
warning: Git tree '/home/sekun/Projects/gnawex' is dirty
warning: Git tree '/home/sekun/Projects/gnawex' is dirty
direnv: renewed cache
direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +DETERMINISTIC_BUILD +HOST_PATH +IN_NIX_SHELL +LD +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_BUILD_CORES +NIX_CC +NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_STORE +NM +OBJCOPY +OBJDUMP +PYTHONHASHSEED +PYTHONNOUSERSITE +PYTHONPATH +RANLIB +READELF +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +_PYTHON_HOST_PLATFORM +_PYTHON_SYSCONFIGDATA_NAME +buildInputs +buildPhase +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +dontAddDisableDepTrack +name +nativeBuildInputs +out +outputs +patches +phases +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system ~PATH ~XDG_DATA_DIRS
sekun@nixos ~/P/gnawex (feature/postgrest)> postgrest
Usage: postgrest [-e|--example] [--dump-config | --dump-schema] FILENAME
  PostgREST 9.0.0 / create a REST API to an existing Postgres database

Available options:
  -h,--help                Show this help text
  -e,--example             Show an example configuration file
  --dump-config            Dump loaded configuration and exit
  --dump-schema            Dump loaded schema as JSON and exit (for debugging,
                           output structure is unstable)
  FILENAME                 Path to configuration file (optional with PGRST_
                           environment variables)

To run PostgREST, please pass the FILENAME argument or set PGRST_ environment
variables.

If you’re wondering how to load the nix shell automatically without running nix develop, look into direnv, and nix-direnv! I’ll probably write about that too since it’s so damn handy that I can’t live without it. It’s like virtualenv on crack.

Conclusion

Could’ve been easier if everything was in the manual. But it isn’t so the entire process of figuring it out involved a lot of Google-fu + man + GitHub code search.

That’s it from me for now. It’s a relatively basic thing to do in Nix since this is in many leagues easier than building a project with it. Still useful though since there’s a lot of pre-built, static binaries out there.

Footnotes