Ask four developers about their operating system, chances are you will get at least five different answers. Just a random sample among friends and colleagues right now would yield Fedora, Ubuntu, Debian, Gentoo, Arch and Mac OS X. Developing software as a team often becomes a challenge of cross-platform compatibility as a result.
Having every developer use the same distro is usually a big ask, so we turn to other solutions of varying degrees of complexity. A little under a decade ago virtualization using Vagrant was the mainstream bet for standardizing the development environment, only to be succeeded by container-based solutions like Docker.
While I personally think that Docker gets a lot of undeserved credit and even on Linux is not the best way of handling containers, it is currently “the standard” which every developer has to interact with sooner or later. It does solve a real need: Being able to specify all the dependencies of a particular piece of software in development and shipping them.
The classical approach
Consider the scenario from a previous article: We have a build.sh or Makefile to build our software, with a good amount of dependencies in tooling and libraries. How do we ensure that every developer in our team and external contributors or newcomers can get started right away?
The classic way of specifying these dependencies is through a README.md file, which will list the dependencies and/or commands to install them, at least on the primary author’s distro. It will usually provide enough information for others to adapt it to their own system, making the latter not a huge issue. Sometimes a project will go the extra mile and provide a shell script to install the dependencies required, with said script occasionally masquerading as a Makefile target or some other build system integration.
Both approaches are not really portable, rarely tested and might go out of date quickly, to say nothing of potential inavailability of the exact same package versions across systems.
The problem is then solved by writing a Dockerfile. It will grow over the course of the project, install and compile all dependencies on a fresh containerized system. The resulting container will run a distribution that no developer uses on their own machine (alpine) but at least it “just works”: Run the docker build command and after a long command-line, a bit of pain mounting the source tree, docker run will launch a shell to build the software. At some point, the Docker launch process will inevitably end up in another shell script though, just for managing the containerized build process.
Getting GUI applications to work requires a lot of extra hoops (example), and there are numerous warts should one try to interact with hardware, or other programs, but at least we do not have to worry about the build failing to run on the new developers desktop because libX is outdated.
Sometimes all of this culminates in the “worst of three worlds”-situation, where there is Dockerfile for the shipped software, a set of instructions to install dependencies in a CI-specific file for continuous integration and developers that still install those manually from instructions found in a README.
This article looks at a way of ultimately unifying all three of these without the use of containers, starting with the local development environment.
Nix and NixOS
In recent years there has been a noticeable uptick in talks about NixOS and the Nix package manager and sometimes they are used interchangeably. The relationship more similar to the one between Debian and dpkg though; NixOS is a distribution built on the Nix package manager.
There is one core difference: nix itself can easily and unintrusively be installed (see the official “Getting Nix” page) on almost any other major Linux distribution. It stores all of its packages under the /nix directory and places only a few symlinks and a $PATH entry elsewhere.
This is what an installation looks like on a fresh debian system (with sudo, curl and xz-utils installed):
alice@debian:/$ curl -L https://nixos.org/nix/install | sh
[...]
performing a single-user installation of Nix...
directory /nix does not exist; creating it by running'mkdir -m 0755 /nix && chown alice /nix' using sudo
copying Nix to /nix/store......................................
installing 'nix-2.3.7'
building '/nix/store/b64sdzfcc4wd2p656i9jskxdmhjkq0cv-user-environment.drv'...
created 6 symlinks in user environment
unpacking channels...
created 1 symlinks in user environment
modifying /home/alice/.profile...
Installation finished! To ensure that the necessary environment
variables are set, either log in again, or type
. /home/alice/.nix-profile/etc/profile.d/nix.sh
in your shell.
We can test this installation by spawning a nix-shell that has figlet installed.
alice@debian:/$ nix-shell -p figlet
[...]
[nix-shell:/]$ figlet nix
_
_ __ (_)_ __
| '_ \| \ \/ /
| | | | |> <
|_| |_|_/_/\_\
Sales pitch
Now that we know that we can install Nix on our system, but why should we actually do so?
Taking a look at Nix, it bills itself as a tool for “reliable and reproducible” package management. In practice it means that Nix will install every program and library into its own prefix under /nix/store, prefixed by a hash of all the inputs going in. This means that an existing application will always link against the same set of libraries, and upgrading any version will create a new instance in the store.
This immediately solves all of our issues with version differences: If every developer has exactly the same software to work with, including of all transitive dependencies and compilers used to build it, there is almost no chance for any differences to crop up.
Since Nix is a package manager and not a container, all these applications integrate neatly into the running system, they can be launched like regular software, provided they’re found in the $PATH, with no VMs or containers involved.
Another big selling point is its declarativeness: Short, maintainable declarations of dependencies instead of build instructions make it easy to describe environments succinctly.
The Nix Package Manager has a long and detailed manual that can be a little intimidating at times, as well as loads of official tutorials, but let’s put it to the test first by building on our previous example.
nix-shell and Nix expressions
Using Nix, the “scripts” to build a package for distribution are called expressions, as they actually are expressions written in a functional programming language. To create a package, a maintainer writes an expression. The output of evaluating this expression using Nix is a derivation, which is a less abstract description of how to build the package.
A derivation can then be built or realized to produce a package. The actual application compilation happens during this step.
For our development environment we are going to pretend that we are creating a proper package for the application under development. This already requires us specifying all dependencies and the build environment. The tool nix-shell is then used to enter a new shell instance where all of these dependencies available, it sits in the middle of the build process for the package we never intend to finish!
Starting out small
We create a shell.nix file in the source code’s root directory, with the following content:
{ pkgs ? import <nixpkgs> { } }:
pkgs.mkShell {
buildInputs = with pkgs; [
gnumake
elmPackages.elm
sassc
cargo
];
}
This expression says that the application we are going to “package” requires the packages gnumake, elmPackages.elm, sassc and cargo.
Once this is done, we can immediately try it out:
$ nix-shell --pure
$ which sassc
/nix/store/1mz9sx8ln0643bgk3cd5wav920sgcgyp-sassc-3.6.1/bin/sassc
$ which make
/nix/store/dp6y0n9cba79wwc54n1brg7xbjsq5hka-gnumake-4.2.1/bin/make
Both sassc and make have been installed in the Nix store and made available via $PATH in our shell instance.
The --pure flag means that we want a “clean” shell: At this point we want no system-wide installed tools to be available inside our new environment, to ensure we are creating an expression that contains all of the dependencies. Later on, when developing, we will lose the --pure flag to keep access to our (but not everyone’s) favorite development tools inside the nix-shell.
We can now try to run our build.sh:
$ ./build.sh
./build.sh: line 14: humblegen: command not found
Looks like we forgot the humblegen CLI app! But it is not available as a Nix package (nix search humblegen does not turn up anything), so what should we do?
Packaging humblegen
We hit a bit of a roadblock — one of our dependencies is not already packaged in Nix. We can solve this problem by packaging it ourselves.
Compared to many other package managers (e.g. dpkg), creating new Nix packages is surprisingly quick. One reason for this is the fact that the required Nix expressions are written in an actual programming language, this makes it very easy to provide plenty of templates in the form of callable functions that are most often good enough.
For our humblegen package we can leverage the buildRustPackage function, which takes a Rust package from anywhere and compiles it. We will save it as humblegen.nix:
{ pkgs }:
pkgs.rustPlatform.buildRustPackage {
name = "humblegen";
version = "0.3.2";
src = pkgs.fetchFromGitHub {
owner = "49nord";
repo = "humblegen-rs";
rev = "690861fddf98e6a71cbd2720b266bebde01c4edb";
sha256 = "1111111111111111111111111111111111111111111111111111";
};
cargoSha256 = "1111111111111111111111111111111111111111111111111111";
}
Note that these hashes are all wrong, we will correct them a bit later. The humblegen.nix expression is not callable without arguments, as it requires at least one parameter: pkgs. We will call it in our shell.nix and add it to buildInputs:
{ pkgs ? import <nixpkgs> { } }:
let humblegen = pkgs.callPackage ./humblegen.nix { inherit pkgs; };
in pkgs.mkShell {
buildInputs = with pkgs; [
humblegen
# ...
Running nix-shell again, we will encounter an error:
hash mismatch in fixed-output derivation '/nix/store/32arw5s8cfgl6s12nw6gdzqdxlcf82sw-source':
wanted: sha256:1111111111111111111111111111111111111111111111111111
got: sha256:1788yaf2c5rl55dk3fmgdb5dwk5yb82wkfw0a7lvsywawj5j8ygr
The hash we gave was incorrect, but Nix calculated the correct one. We could have downloaded and determined it manually using nix-hash, but in this case we trust the HTTPS connection used for downloading and simply copy it into our expression. Our reward is another hash error:
hash mismatch in fixed-output derivation '/nix/store/9d5vnpjnydnpdrwphp3g9609pkdyrfj5-humblegen-vendor':
wanted: sha256:1111111111111111111111111111111111111111111111111111
got: sha256:06wsk2i24a40zj54w49q0671qhr6rmf7k4d5fwslqvbsd00qkpn4
This is the hash of the Cargo.lock lockfile the application itself.
While it may strange to just copy over these hashes, they will protect us in the future against malicious or accidental changes, provided we are sure that at this point in time our downloads are trustworthy.
Running nix-shell a third time, we can see some compiling going on, before humblegen throws us another curveball:
error[E0433]: failed to resolve: could not find `primitive` in `std`
--> humblegen-rt/src/serialization_helpers.rs:51:10
|
51 | std::primitive::str::parse(query)
| ^^^^^^^^^ could not find `primitive` in `std`
This one is a bit tougher and not Nix’s fault: The std::primitive was only added in Rust 1.43, which has not made it into most stable distros so far!
This would put a damper on our plans with some other distributions; as mixing stable and unstable packages can result in a mess. With Nix, we do not have to fear anything though, as multiple versions of packages can coexist, complete with a separate set of libraries to go with them.
First, we need to get access to unstable though. All Nix channels are hosted on GitHub, so we can retrieve them in the same manner as the humblegen source itself.
let
unstable = import (pkgs.fetchFromGitHub {
owner = "NixOS";
repo = "nixpkgs";
rev = "83182f4936e848848428559ace9ab2b627350998";
sha256 = "1dql3mzmshnyd23sxxlsp1h9i6wcmg5dpqj6lw4m8rirai9yh123";
}) { };
humblegen = pkgs.callPackage ./humblegen.nix { pkgs = unstable; };
in pkgs.mkShell {
# ...
The resulting unstable is a collection of packages in the same vein as the local <nixpkgs>, but pinned at the specific commit. We pass it to our already written humblegen.nix expression (instead of inheriting pkgs) and run nix-shell again, only to be greeted by another hash mismatch:
hash mismatch in fixed-output derivation '/nix/store/ygad1q11qncnpy9nybsvba85r0i9inkz-humblegen-vendor.tar.gz':
wanted: sha256:06wsk2i24a40zj54w49q0671qhr6rmf7k4d5fwslqvbsd00qkpn4
got: sha256:1qcra34f4n1by19vk3p8vr4fi15qqk8d0qlgpia6hrhhhxprjc9q
cannot build derivation '/nix/store/gdqyw2a60055phk8gf687dqq04mvfhaa-humblegen.drv': 1 dependencies couldn't be built
The hash changed again when we changed the toolchain, if we were to dig a little deeper, we would find out that this is likely due to a change in cargo vendor, as Nix hashes the complete package with all dependencies to create its own hash.
We fix the hash, run nix-shell --pure and are finally able to run ./build.sh without issues with the following derivation:
{ pkgs ? import <nixpkgs> { } }:
let
unstable = import (pkgs.fetchFromGitHub {
owner = "NixOS";
repo = "nixpkgs";
rev = "83182f4936e848848428559ace9ab2b627350998";
sha256 = "1dql3mzmshnyd23sxxlsp1h9i6wcmg5dpqj6lw4m8rirai9yh123";
}) { };
humblegen = pkgs.callPackage ./humblegen.nix { pkgs = unstable; };
in pkgs.mkShell {
buildInputs = with pkgs; [
humblegen
gnumake
elmPackages.elm
sassc
cargo
];
}
While this may seem a bit complicated, we can look at what we have achieved so far: With a single command, we offer up a development environment that uses an unreleased but pinned version of a Rust tool compiled with the most recent available Rust compiler version, all while using tools for other languages from the stable repository — and we are guaranteed no conflicts and can ship this in ~ 30 lines of code total.
Note that over time, the complexity of this is going to reduce a bit, as we no longer need to access the unstable repository.
Going the extra mile
For a little extra flourish, we can finalize our shell.nix like this:
{ pkgs ? import <nixpkgs> { } }:
let
stable = import (pkgs.fetchFromGitHub {
owner = "NixOS";
repo = "nixpkgs-channels";
rev = "b50d55871fb7de1e5791bbd56738ff20f4d15f2c";
sha256 = "0sxkpacyzpmc5n658pj287j6bd7sc2d6r1azyrpmd5fyx8q1ihvs";
}) { };
unstable = import (stable.fetchFromGitHub {
owner = "NixOS";
repo = "nixpkgs";
rev = "83182f4936e848848428559ace9ab2b627350998";
sha256 = "1dql3mzmshnyd23sxxlsp1h9i6wcmg5dpqj6lw4m8rirai9yh123";
}) { };
humblegen = stable.callPackage ./humblegen.nix { pkgs = unstable; };
in stable.mkShell {
buildInputs = with stable; [
humblegen
gnumake
elmPackages.elm
sassc
unstable.cargo
];
}
We pinned the stable packages as well, so now all of our devs are guaranteed to have the same set of tools, even if they use a very different Nix version.
Finally we install unstable.cargo instead of the version from stable since our code will at some point probably require the features from the latest compiler due to it using humblegen.
Conclusion
We have fixated all our build dependencies including tooling in two Nix derivations that we can ship to anyone with a reasonably recent Nix package manager version and expect for them to have a running build environment. It also handles a rather thorny unpackaged Rust application that we compile directly from source and seamlessly integrates it into our build process.
Some unwieldy hashes aside, the Nix expressions are also very readable, and we have barely scratched the surface of what is possible here.
In the future, we will show how this makes CI integration a breeze.