Welcome to the 9th Nix pill. In the previous 8th pill we wrote a generic builder for autotools projects. We feed build dependencies, a source tarball, and we get a Nix derivation as a result.
Today we stop by the GNU hello world program to analyze build and runtime dependencies, and enhance the builder in order to avoid unnecessary runtime dependencies.
Let's start analyzing build dependencies for our GNU hello world package:
$ nix-instantiate hello.nix /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv $ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv /nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.10.tar.gz /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv /nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv /nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv /nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh /nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv /nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv /nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv /nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv /nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv /nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv /nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv
It has exactly the derivations referenced in the derivation
function, nothing more, nothing less. Some of them might not be used at
all, however given that our generic mkDerivation function always pulls
such dependencies (think of it like
build-essential
of Debian), for every package you build from now on, you will have these
packages in the nix store.
Why are we looking at .drv files? Because the hello.drv file is the representation of the build action to perform in order to build the hello out path, and as such it also contains the input derivations needed to be built before building hello.
NAR is the Nix ARchive. First question: why not tar? Why another archiver? Because commonly used archivers are not deterministic. They add padding, they do not sort files, they add timestamps, etc.. Hence NAR, a very simple deterministic archive format being used by Nix for deployment. NARs are also used extensively within Nix itself as we'll see below.
For the rationale and implementation details you can find more in the Dolstra's PhD Thesis.
To create NAR archives, it's possible to use
nix-store --dump and
nix-store --restore. Those two commands work
regardless of /nix/store.
Something is different for runtime dependencies however. Build
dependencies are automatically recognized by Nix once they are used in
any derivation call, but we never specify what are the
runtime dependencies for a derivation.
There's really black magic involved. It's something that at first glance makes you think "no, this can't work in the long term", but at the same it works so well that a whole operating system is built on top of this magic.
In other words, Nix automatically computes all the runtime dependencies of a derivation, and it's possible thanks to the hash of the store paths.
Steps:
Dump the derivation as NAR, a serialization of the derivation output. Works fine whether it's a single file or a directory.
For each build dependency .drv and its relative out path, search the contents of the NAR for this out path.
If found, then it's a runtime dependency.
You get really all the runtime dependencies, and that's why Nix deployments are so easy.
$ nix-instantiate hello.nix /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv $ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello $ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19 /nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3 /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
Ok glibc and gcc. Well, gcc really should not be a runtime dependency!
$ strings result/bin/hello|grep gcc /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64
Oh Nix added gcc because its out path is mentioned in the "hello" binary. Why is that? That's the ld rpath. It's the list of directories where libraries can be found at runtime. In other distributions, this is usually not abused. But in Nix, we have to refer to particular versions of libraries, thus the rpath has an important role.
The build process adds that gcc lib path thinking it may be useful at runtime, but really it's not. How do we get rid of it? Nix authors have written another magical tool called patchelf, which is able to reduce the rpath to the paths that are really used by the binary.
Even after reducing the rpath, the hello binary would still depend upon gcc because of some debugging information. This unnecesarily increases the size of our runtime dependencies. We'll explore how strip can help us with that in the next section.
We will add a new phase to our autotools builder. The builder has these phases already:
First the environment is set up
Unpack phase: we unpack the sources in the current directory (remember, Nix changes dir to a temporary directory first)
Change source root to the directory that has been unpacked
Configure phase: ./configure
Build phase: make
Install phase: make install
We add a new phase after the installation phase, which we call
fixup phase. At the end of the
builder.sh follows:
find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null
That is, for each file we run patchelf --shrink-rpath
and strip. Note that we used two new commands here,
find and patchelf.
Exercise: These two
deserve a place in baseInputs of
autotools.nix as findutils and
patchelf.
Rebuild hello.nix and...:
$ nix-build hello.nix [...] $ nix-store -q --references result /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19 /nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello
...only glibc is the runtime dependency. Exactly what we wanted.
The package is self-contained, copy its closure on another machine and
you will be able to run it. Remember, only a very few components under
the /nix/store are required to
run nix.
The hello binary will use that exact version of glibc library and
interpreter, not the system one:
$ ldd result/bin/hello linux-vdso.so.1 (0x00007fff11294000) libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000) /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000)
Of course, the executable runs fine as long as everything is under the
/nix/store path.
Short post compared to previous ones as I'm still on vacation, but I hope you enjoyed it. Nix provides tools with cool features. In particular, Nix is able to compute all runtime dependencies automatically for us. This is not limited to only shared libraries, but also referenced executables, scripts, Python libraries etc..
This makes packages self-contained, ensuring (apart data and configuration) that copying the runtime closure on another machine is sufficient to run the program. That's why Nix has one-click install, or reliable deployment in the cloud. All with one tool.
...we will introduce nix-shell. With nix-build we always build derivations from scratch: the source gets unpacked, configured, built and installed. But this may take a long time, think of WebKit. What if we want to apply some small changes and compile incrementally instead, yet keeping a self-contained environment similar to nix-build?