In this weeks issue
- Configuration control with NixOS and Kairos
- Tools: blxrep, udpspeeder, Node-RED and below
- Random: LTE/5G vulnerabilities and Mr. Beast
Configuration control, NixOS and Kairos
A client asked me about configuration control and brought up Ansible, Puppet and so on. I’m not sure if configuration control is the right term, but given the context my client was asking in, I immediately thought of NixOS and Kairos. Although these two are very different there are some commonalities:
- Unified declarative configuration, meaning you can configure your whole OS using the same syntax (although you may have to add support for some settings)
- Immutable, meaning you don’t get configuration drift at runtime (with caveats)
- Atomic updates and rollback, meaning if an update goes wrong the system will be fine and you can select previous generations at boot
There is quite a large group of software that comes into the
configuration management and infrastructure as code (IaC) categories
that will control the state of a Linux installation. The idea being
that you can automate installing and updating one or more Linux
devices. There are a number of systems which can install and
configure Ubuntu, these act as an external controller and usually
use an agent, such as sshd, to control the box once it
is up and running. During installation something like cloud-init can
be used to get the initial configuration.
This isn’t exactly how NixOS and Kairos work. NixOS is in fact a stand alone Linux distribution which is built upon the Nix language and package manager. Kairos is a meta-distribution which transforms other distributions into immutable systems with generations and rollback. They both allow creating customized ISO and VM images.
Kairos uses container technologies to take a distribution like Ubuntu and turn it into an immutable OS. The resulting image still uses the Ubuntu kernel and to my knowledge there is no container runtime. However it uses file system overlays at runtime to allow rollback. At build time it uses Dockerfiles and so it’s essentially using Dockerfiles to package a bare metal OS. The overall system configuration is done using a cloud-init file in a similar fashion to how NixOS uses Nix to configure everything (see below). The cloud-init syntax can be extended using bundles, which themselves are created with Dockerfiles and are kind of like meta-packages.
Nix is actually a functional programming language that is intended for reproducibly building and packaging software. You can install the Nix package manager on Linux, Mac and maybe Windows at some point. It’s in fact the largest and most up to date OSS repository in the world by a very long way. Only AUR comes close and that is actually still less fresh than a NixOS stable release from over 3 years ago. The Google and Apple app stores are larger, but Nix is catching up.
NixOS is a Linux distribution built with Nix. Almost everything
on NixOS can be configured from a single Nix file. The base Nix
system is partially immutable and supports rolling back to previous
generations. I started using NixOS because I was fed up of
reinstalling and configuring machines from scratch as well as
maintaining lots of different config files. I also loved the Nix
package manager, especially because I could just type
nix run nixpkgs#some-program or
nix shell nixpkgs#some-program and try out some program
without installing it or using a container.
NixOS also has the steepest learning curve I’ve ever encountered in a Linux distribution. You aren’t just learning an unusual distribution, you are also learning an unusual programming language and its libraries. A lot of software does stupid stuff like download binaries that link to a specific libc in a specific location. This does not work with Nix unless you implement a workaround and this is usually not that easy. The fact then that NixOS has more than just a few users is a testament to how incredibly powerful it is. There has to be a strong incentive to use this software because it is bloody difficult to get into it.
I vaguely remember a post from someone saying “I finally wrote my first Nix flake/package and it only took me two years!”. Obviously the story is quite different with Kairos which still requires some mental adjustments, but will be more familiar to a lot of system admins who have already been exposed to Kubernetes, cloud-init and whatever distro they decided to wrap.
It could actually make sense to wrap NixOS with Kairos because the level of immutability is different between them, but there are a lot of details to sort through before deciding something like that. Indeed there is a lot of depth to this topic.
Relevant links
- https://nixos.org/
- https://kairos.io/
Tools
- blxrep - disk replication tool that uses eBPF tracepoints
- udpspeeder - Wraps UDP packets (e.g. VPN traffic) providing forward error correction codes making any UDP traffic more reliable
- Node-RED - Low code event-driven programming. This is great for writing programs that you can then hand off to non-coders, especially electrical and mechanical engineers who despise anything text based. It has a module for Modbus which I covered in my previous newsletter.
- below - Like top, but for CGroups