Static or Dynamic Linking: What Changes in the Binary and at Startup

Investigation · Part 6 of 20 · From source to a running process

A statically linked program is supposed to need nothing from the system at runtime. Right? Here is my static C program, looking up a user name that does not exist on the machine:

$ strace -e trace=openat ./whoami_static nobody-here
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/libnss_systemd.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/ld-linux-x86-64.so.2", O_RDONLY|O_CLOEXEC) = 3
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x30} ---
+++ killed by SIGSEGV (core dumped) +++

A static program opened five shared objects, among them the shared libc and the dynamic loader, and then it crashed. This one surprised me a lot. I come back to it at the end, because it says the most about static linking. First the normal differences, measured on the same laptop as in the first part of this series.

Five builds of hello world

I built the hello world of the first part in five ways: gcc, gcc -static, gcc -static-pie, rustc -O, and rustc -O -C target-feature=+crt-static. I was not sure that the last one works with glibc, but it does. A musl target, the other usual way to make a static Rust binary, is not installed on this machine (rustup target list --installed shows only x86_64-unknown-linux-gnu), so I stayed with glibc.

$ wc -c hello_c_dyn hello_c_static hello_c_static_pie hello_rs hello_rs_static
   15896 hello_c_dyn
  778440 hello_c_static
  819904 hello_c_static_pie
 4507728 hello_rs
 5463200 hello_rs_static
11585168 total
...
$ ldd ./hello_c_dyn
	linux-vdso.so.1 (0x00007ffdee5d2000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007fe247d76000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fe247f9c000)
$ ldd ./hello_c_static
	not a dynamic executable
$ ldd ./hello_c_static_pie
	statically linked

The static C file is 49 times bigger, because it contains the parts of glibc it uses. The static Rust file grows by less than 1 MB, because the Rust standard library was already inside the dynamic one (the second part showed that most of the Rust file is debug information of std). file tells the important difference between the two static C files:

hello_c_static: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 4.4.0, not stripped
hello_c_static_pie: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, for GNU/Linux 4.4.0, not stripped

-static gives a classic executable at a fixed address. -static-pie gives a PIE without an interpreter: it still has a DYNAMIC segment, because it must apply its own relocations at startup, but no INTERP line, so the kernel starts it directly. The Rust static build is a static PIE too.

The start

In the first part I counted the system calls between execve and the write of the greeting. Here are the five programs:

system calls after execve and before the write of the greeting:
  hello_c_dyn           32
  hello_c_static        14
  hello_c_static_pie    14
  hello_rs              61
  hello_rs_static       32

The static Rust program loses all the loader work: no cache, no libraries to open and map. What is left is libc's own setup and the Rust runtime setup. Then I timed 1000 starts of each program with the launcher of the first part, pinned to one performance core, with a load average around 3.5 and other experiments running the same evening, some of them on this core:

program          starts    median       p10       p90       min       max  faults
hello_c_dyn        1000    272 us    262 us    374 us    252 us   1012 us      63
hello_c_static     1000    132 us    126 us    149 us    124 us   1046 us      31
hello_c_static_pie   1000    136 us    132 us    148 us    129 us    995 us      31
hello_rs           1000    417 us    392 us    458 us    378 us   1251 us      81
hello_rs_static    1000    193 us    181 us    209 us    171 us   1310 us      42

(Each line is a separate run of 1000 starts of one program, and the script prints the five lines together at the end.) Static C starts in half the time of dynamic C (in the run where the five programs alternate, 176 against 300 microseconds, a bit more than half). For Rust the gain is bigger, 417 down to 193 microseconds, because the dynamic Rust program loads two libraries. The static PIE costs almost nothing more than the classic static file, its self relocation is small. The page faults follow the same order: a static program touches fewer pages at startup.

The memory

The start is faster, but there is a cost on the other side. A dynamic program shares the pages of libc with every other process that uses libc. A static program carries its own copy. To see this I made a program that only calls pause(), built it both ways, started one copy and then three copies of each, and read /proc/PID/smaps_rollup:

$ sh memory.sh 1
sleeper_dyn, 1 running, /proc/830931/smaps_rollup:
Rss:                1360 kB
Pss:                 108 kB
Shared_Clean:       1256 kB
Shared_Dirty:          0 kB
Private_Clean:        12 kB
Private_Dirty:        92 kB
sleeper_static, 1 running, /proc/830932/smaps_rollup:
Rss:                 648 kB
Pss:                 644 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:       588 kB
Private_Dirty:        56 kB

Rss counts every page of the process that is in RAM. Pss, the proportional set size, divides each page by the number of processes that share it. So Pss is a fair answer to "how much memory does this process cost alone".

The dynamic sleeper has the bigger Rss, 1360 kB, but a Pss of only 108 kB. Its libc pages are counted in Rss, and almost nothing in Pss:

sleeper_dyn, the libc.so.6 lines of /proc/830931/smaps:
  r--p libc.so.6  Rss  144 kB  Pss   0 kB
  r-xp libc.so.6  Rss  832 kB  Pss   2 kB
  r--p libc.so.6  Rss   60 kB  Pss   1 kB
  r--p libc.so.6  Rss   16 kB  Pss  16 kB
  rw-p libc.so.6  Rss    8 kB  Pss   8 kB

832 kB of libc code in RAM, and this process pays 2 kB for it. So the same pages are mapped by hundreds of other processes on my laptop. Only the last two lines are really its own, the data of libc after relocation and the writable data.

The static sleeper is the opposite. Rss 648 kB, Pss 644 kB, almost everything private: 588 kB of its own copy of glibc code that nobody else maps. With three copies running, these pages are shared between the three, and the Pss of one copy goes down to 256 kB:

sleeper_static, 3 running, /proc/830950/smaps_rollup:
Rss:                 652 kB
Pss:                 256 kB
Shared_Clean:        592 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        60 kB

So a static program costs more memory, except when many copies of exactly the same file run. (A detail I had to fix during the experiment: just after the build, the pages of the new files were still dirty in the page cache, and smaps counted them as dirty. The script now calls sync on the two files before starting them.)

What a static binary still needs

Back to the crash. At link time, GNU ld had printed a warning for the static build:

$ gcc -O2 -static whoami.c -o whoami_static
/usr/bin/ld: whoami.o: in function `main':
whoami.c:(.text.startup+0x15): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
$ grep -E '^passwd' /etc/nsswitch.conf
passwd: files systemd

The answer is in /etc/nsswitch.conf. glibc looks up users through NSS, the name service switch, and the passwd line says: first the files service (/etc/passwd), then the systemd service. The nsswitch.conf(5) man page says that a service is a shared library called libnss_SERVICE.so.X. The files code is inside libc, so the static program has its own copy, and for root it found the answer in /etc/passwd and never needed anything else:

$ ./whoami_static root
root has uid 0
$ ./whoami_static nobody-here
exit status 139

For nobody-here, files finds nothing, and glibc goes to the next service. To use systemd it must dlopen libnss_systemd.so.2, and this library depends on the shared libc and on the loader. So the static process loaded a second C library and a second dynamic loader into itself. gdb shows where it stopped:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7f64516 in _nss_systemd_is_blocked () from /usr/lib/libnss_systemd.so.2
#0  0x00007ffff7f64516 in _nss_systemd_is_blocked () from /usr/lib/libnss_systemd.so.2
#1  0x00007ffff7f6fb68 in _nss_systemd_getpwnam_r () from /usr/lib/libnss_systemd.so.2
#2  0x000000000041b8ac in getpwnam_r ()
#3  0x000000000041b6ef in getpwnam ()
#4  0x00000000004030a9 in main ()
...
   0x00007ffff7f6450f <+15>:	call   *0x657b3(%rip)        # 0x7ffff7fc9cc8
   0x00007ffff7f64515 <+21>:	pop    %rbp
=> 0x00007ffff7f64516 <+22>:	mov    0x30(%rax),%eax
...
rax            0x0                 0

the call before the crash goes through the pointer at 0x7ffff7fc9cc8, which points to:
__tls_get_addr in section .text of /usr/lib/ld-linux-x86-64.so.2

The function calls __tls_get_addr to find a thread-local variable of the library, gets 0 back, and reads 0x30 bytes after it. This is the si_addr=0x30 of the strace output. The __tls_get_addr here belongs to the second loader, the shared one, which was not the loader that set up this thread. Thread-local storage has its own part later in the series, here it is enough to see that it did not work in this mix.

And then the versions:

$ pacman -Qo /usr/lib/libc.a /usr/lib/libc.so.6 /usr/lib/libnss_systemd.so.2
/usr/lib/libc.a is owned by glibc 2.43+r5+g856c426a7534-1
/usr/lib/libc.so.6 is owned by glibc 2.43+r5+g856c426a7534-1
/usr/lib/libnss_systemd.so.2 is owned by systemd-libs 259.1-1

In fact, the static libc and the shared libc come from exactly the same package. The condition of the warning was true, and it still crashed. The dynamic version of the same program has no problem: it prints "nobody-here: no such user".

Rust: the same crash, without the warning

I made the same test in Rust, with getpwnam declared in an extern "C" block and the +crt-static build:

$ rustc --edition 2024 -O -C target-feature=+crt-static whoami.rs -o whoami_rs_static  (what it printed:)
$ ./whoami_rs_static root
root: found
$ ./whoami_rs_static nobody-here
exit status 139

The same crash, and the build printed nothing at all. Where did the warning go? It is not a feature of the compiler. The text is stored in libc.a itself, in a special section of the object that contains getpwnam:

$ ar x /usr/lib/libc.a getpwnam.o && readelf -p .gnu.warning.getpwnam getpwnam.o

String dump of section '.gnu.warning.getpwnam':
  [     0]  Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

GNU ld found this section and printed its text when my program used getpwnam. rustc links with LLD, and when I linked the C program with LLD too (gcc -static -fuse-ld=lld), it printed nothing either. So with this toolchain, a static Rust build that uses NSS gets no warning at all.

How I would choose

Static linking gave me a faster start (half the time here), one file that does not depend on the loader or the libc of the target, and no dynamic relocations against other libraries. The price:

a bigger file, and memory that is not shared with other programs;
a security fix in glibc helps a dynamic program at its next start, but a static program only after a rebuild;
LD_PRELOAD and other tricks of the dynamic loader have nothing to act on (a later part shows this);
and the features of glibc that load modules at runtime, like NSS for users, groups and host names, still need the shared libraries, and in my test they crashed the program.

So for a glibc program that looks up users or hosts, "static" is not really static. For a small tool that only reads files and writes output, it can be a good choice. I would test the exact lookups that the program makes, with users and names that are not in the local files, before trusting a static build.