Compiler says no!

Herding cats with NixOS

Instead of taking your servers behind the barn and shooting them with Kubernetes, try a little more loving care with nixos-rebuild.

Exploring trends of the past seems to be a theme this quarter for me, so it is only apt I arrived at the old Pets vs Cattle argument, which was meant to illustrate the replaceability of a cattle server that can be killed and replaced with low impact to the service, in contrast to the pet server for which the vet on pager duty will be woken at any hour if it even so much as coughs.

Nowadays I would state three core aspects of a pet server: it's non-disposable1, "hand-fed", i.e. built without automation, and there are few of its kind. The non-automated part is a bit of a straw-man these days, not many people hand-craft servers in production anymore, and if you consider ephemeral containers the default delivery artifact, the "disposable" box is ticked more often than not, too2.

A contrasting, recurrent message is a few boxen being powerful enough for the majority of use-cases already, if you are not in the business of reselling compute and/or storage resources3. If not, consider taking a look at your software's performance characteristics4!

I like to combine these two aspects on mid-sized projects, which puts me into the business of herding cats5: A small number of servers with personality that suffice to do all the work and only sometimes demand attention, but still come with all the modern declarative, reproducible and mostly ephemeral characteristics. At that point Kubernetes and its k3s-shaped friends are the wrong tool though, especially on cheap VPS machines with 1 GB of RAM, where kubelet and the control plane leave few resources for your application.

Becoming a cat owner

Running NixOS on your desktop may not be mainstream just yet and would be deserving of a proper intro, but we will sum it up as "a way to describe an entire system declaratively" for now: in a flake-based setup, /etc/nixos/flake.nix ties together settings such as installed packages, the desktop environment or console keyboard layout.

A system defined in this manner can be activated through nixos-rebuild switch --flake /etc/nixos#my-desktop, which builds the configuration and restarts a few services to make it take effect. This entire process is nearly atomic and the switchover close to instant.

The interesting part is realizing that servers can just as well be defined in flakes:

{
  nixosConfigurations.leibniz = nixpkgs.lib.nixosSystem {
    system = "x86_64-linux";

    modules = [
      nixdrawer.nixosModules.hetzner-cloud
      {
        networking = {
          hostName = "leibniz";
          firewall.allowedTCPPorts = [ 6667 ];
        };

        services.inspircd = {
          enable = true;
          config = ''
            <server name="leibniz.example.com"
                    description="Dance IRC server"
                    network="DanceNet">
            <bind address="" port="6667" type="clients">
            <connect name="main" allow="*">
          '';
        };
      }
    ];
  };
}

With a little help from my Hetzner cloud Nix module, this is enough to describe a working IRC server running InspIRCd.

The fundamental realization is that nixos-rebuild can activate that configuration on an existing NixOS machine over SSH, but we need to bootstrap it first. The aptly named nixos-anywhere can kexec into a temporary system, apply the disk layout from our configuration and install the system:

$ nix run github:nix-community/nixos-anywhere -- \
    --flake .#leibniz --copy-host-keys \
    root@leibniz.example.com

Our declarative partitioning configuration hidden in hetzner-cloud.nix and nixos-anywhere is smart enough to invoke its destructive side. This already leaves us with a deployed system, subsequent updates only need nixos-rebuild:

$ nixos-rebuild switch --flake .#leibniz --target-host root@leibniz.example.com

It looks fairly inconspicuous, but this will build the host locally, upload it to the remote machine and activate it. nixos-rebuild can also delegate the build with --build-host, which is useful when the local machine cannot build for the target architecture6. A third machine can thus act as a build server:

$ nixos-rebuild switch --flake .#leibniz \
    --build-host root@builder.example.com \
    --target-host root@leibniz.example.com

No CI/CD setup7, no fancy gitops, no GUIs to click for deployments. The flake.nix along with a flake.lock gets checked into your repo, and the server gets reproducibly updated.

Automated cat feeder

I recently released dance, a small ops tool8 to lean into this way of deploying things. It saves me from having to remember the precise nixos-rebuild commands, but even more importantly which host definition goes where. It uses an inventory file:

[hosts.www]
ssh_host = "www.example.com"
flake_path = "www"
flake_name = "www"

[hosts.prty]
ssh_host = "party.example.com"
flake_path = "ex.com"
flake_name = "prty"

A single dance status (a -i inventory.toml is implied) will then let you know what's up:

marc@marc-desktop /c/infra (master)_% dance status
+------+--------------+------------+------+--------------------+
| ID   | FLAKE       | STATUS      | VALD | CHANGE             |
+------+-------------+-------------+------+--------------------+
| www  | www#www     | out of date | yes  | 1hdsxqp6->p1jaaqy7 |
+------+-------------+-------------+------+--------------------+
| prty | ex.com#prty | out of date | yes  | jl4lng03->4qvqv0bf |
+------+-------------+-------------+------+--------------------+

Flake builds are (mostly) deterministic, so nix can determine the hash of the system solely from the inputs and the flake. Remote hosts also store the hash of the active system, so we have something to compare to.

A single dance deploy will concurrently acquire a lock on each remote machine, build the system locally and switch it over. It also starts an automatic watchdog timer on the remote that resets it to the previous generation if not cleared, in case you happened to mess up your SSH configuration.

If you have ever used Nix-based servers with LLM agents, you will notice they are a very good fit: everything is declarative in a few text files, and the outputs are testable: an agent will use NixOS's build-vm target without hesitation to boot the described machine in a QEMU VM and white-box test it.

Testing production systems is often a different beast though, credentials and proper setup become more important. Dance addresses this by allowing the definition of verifiers, like this one that checks whether each site came up with working HTTPS:

{
  danceVerifiers.${system}.www = pkgs.writeShellApplication {
    name = "verify-www";
    runtimeInputs = [ pkgs.hurl ];
    text = ''
      hurl --test <<'EOF'
      GET https://compilersaysno.com
      HTTP 200

      GET https://49nord.de
      HTTP 200
      EOF
    '';
  };
}

Verifiers can be simple smoke tests, but you are free to make them as complex as you like. dance will run them after each deployment against the production server and automatically roll back if things fail.

An entire litter

Dance's contribution is rather small, as the impressive features exist thanks to nix and the tireless maintainers that make it work in the first place. Let's leverage them further to make managing a small fleet9 of machines straightforward as well:

Imagine we wanted to bring up a cluster of 10 IRC servers, avoiding Big Tech this time and getting us some Hetzner. Conveniently there's an hcloud tool we use for one-time provisioning:

$ hosts="leibniz kant hegel marx nietzsche heidegger frege \
wittgenstein schopenhauer adorno"
$ for host in $hosts; do
    hcloud server create --name "$host" --type cx23 \
      --image debian-13 --location nbg1 --ssh-key marc
  done

The cattle ranchers will point out that we are still setting up servers "by hand" instead of adding a provisioner slinging our credit card that can create resources at will. I find this straightforward, especially given that it only happens a few times a year.

Now let's add some subdomains for the addresses assigned by Hetzner:

$ for host in $hosts; do
    hcloud zone rrset create --name "$host" --type A --ttl 60 \
      --record "$(hcloud server ip "$host")" example.com
  done

Once the machines are up, we are now the proud owners of ten Debian machines with our SSH key installed.

The test flake is longer once TLS and verification are included, but the part that turns one IRC server configuration into ten is small:

let
  inherit (nixpkgs) lib;
  domain = "example.com";
  hub = "leibniz";
  linkPassword = "dance-test-server-link";
  hosts = [
    "leibniz"
    "kant"
    "hegel"
    "marx"
    "nietzsche"
    "heidegger"
    "frege"
    "wittgenstein"
    "schopenhauer"
    "adorno"
  ];
  fqdn = hostName: "${hostName}.${domain}";
  peersFor = hostName: if hostName == hub then lib.remove hub hosts else [ hub ];
  mkIrcConfig = import ./inspircd-config.nix { inherit lib; };
  ircConfig =
    hostName:
    mkIrcConfig {
      serverName = fqdn hostName;
      hubName = fqdn hub;
      peers = map fqdn (peersFor hostName);
      inherit linkPassword;
    };
  mkServer =
    hostName:
    lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        nixdrawer.nixosModules.hetzner-cloud
        {
          networking = {
            inherit domain hostName;
            firewall.allowedTCPPorts = [
              80
              6697
              7000
            ];
          };
          security.acme = {
            acceptTerms = true;
            certs.${fqdn hostName} = {
              listenHTTP = ":80";
              reloadServices = [ "inspircd.service" ];
            };
          };
          systemd.services.inspircd = {
            after = [ "acme-finished-${fqdn hostName}.target" ];
            wants = [ "acme-finished-${fqdn hostName}.target" ];
            serviceConfig.SupplementaryGroups = [ "acme" ];
          };
          services = {
            inspircd = {
              enable = true;
              config = ircConfig hostName;
            };
            openssh.enable = true;
          };
          users.users.root.openssh.authorizedKeys.keyFiles = [ ./marc.pub ];
        }
      ];
    };
in
{
  nixosConfigurations = lib.genAttrs hosts mkServer;
}

By writing a small mkServer function to generate each server and having genAttrs call it for each hostname, we get a nice, reusable definition for each member of our IRC network. All this without generating YAML manifests from string templates, which is a terrible idea!

security.acme provides nice Let's Encrypt support out of the box10. Our configuration-generating function needs to grow some TLS support, too:

<bind address="" port="6697" type="clients" sslprofile="tls">
   <bind address="" port="7000" type="servers" sslprofile="tls">
   <module name="ssl_gnutls">
   <module name="spanningtree">
   <sslprofile name="tls" provider="gnutls"
               certfile="/var/lib/acme/${serverName}/fullchain.pem"
               keyfile="/var/lib/acme/${serverName}/key.pem">

While the integration is more work than we'd like11, the easier TLS administration should make up for it.

The same verifier mechanism checks each member of the network by connecting to port 6697 over TLS. Because verifiers receive DANCE_HOST_ID, one verifier definition can cover every host.

Dance expects its targets to already run NixOS, so we first repeat the one-time bootstrap for every Debian machine:

$ for host in $hosts; do
    nix run github:nix-community/nixos-anywhere -- \
      --flake ".#$host" --copy-host-keys \
      "root@$host.example.com"
  done

The last thing we need is an inventory file12 for our litter:

[hosts.leibniz]
ssh_host = "leibniz.example.com"
flake_path = "."
flake_name = "leibniz"

# Repeat for the remaining hosts.

With all this done, our reward is getting to watch a little TV of the system doing its thing every time we make a change to the system:

Download terminal recording

Conclusion

The whole pets vs cattle argument often presents a false dichotomy. At medium scale, it is entirely reasonable to put a little more love into server administration without sacrificing good practices like declarative infrastructure as code. Nix is what stands out here; Dance is just the cherry on top, and the approach works fine without it. If you do not believe me, count how many separate, established tools an equivalent setup would ordinarily involve, and how much of their work the Nix ecosystem covers here.

And, more importantly, medium scale should be enough, right?

  1. To be fair, in the linked original source for pets vs cattle, this was the only hard requirement to be considered a pet. ↩

  2. At least until someone inevitably has to do the hard part of storing the actual data, for which a big SaaS provider will eventually be employed before everyone congratulates themselves for doing a good job of creating such a well-architected, fault-tolerant system to serve the entirety of their 5000 users. ↩

  3. I used to point out that prices have come down, too, but that was before AI ate all the RAM and even cheap VPS instances quadrupled in price. ↩

  4. Feel free to get in touch if you want me to help you do that! ↩

  5. I may be underqualified to write about cats as the owner of a single 140 lb mountain of fur, who is more likely to bark at the former, but rest assured, the four-legged co-owner of this post was petted frequently during the writing process. ↩

  6. It is entirely possible that I am not doing Nix's supposedly excellent cross-compilation capabilities justice here, I just did not have the chance to try them yet. ↩

  7. You could, of course. I am not stopping you! ↩

  8. A (now former!) friend recently hurt my feelings saying "oh, so you made Ansible for nix!". ↩

  9. Yes, I am aware of the irony after having advocated against that only a few paragraphs earlier. ↩

  10. Raise your hand if you have ever had to fix cert-manager after it mysteriously broke on its own. ↩

  11. The InspIRCd module is relatively bare-bones and accepts the daemon configuration verbatim. More comprehensive NixOS service modules expose typed options that avoid this XML-generating detour, alongside conveniences such as services.<name>.openFirewall and appropriate systemd ordering. ↩

  12. One could argue that it might be a good idea to also use a flake/nix instead of TOML, we'll leave that for a later version of dance though. ↩