Mehdi Akiki
Published on

What Happens When You Run a Program: From execve to main

Authors
  • Mehdi Akiki avatar
    Name
    Mehdi Akiki
    Twitter

Investigation · Part 1 of 20 · From source to a running process

My C hello world has one line in main. I ran it under strace and counted. Between the execve that starts the process and the write that prints the line, there are 32 other system calls. The same program linked statically makes 14. The Rust version makes 61.

Most of these calls happen before my code runs at all. So who makes them? And why does each one have to happen before main?

I made an experiment on my laptop to answer this. For every system call I looked for the code that made it: the kernel, the dynamic loader, the C library or the Rust runtime. At the end I timed one start of each program. The programs, and a script that regenerates every output in this article, are in the fixture of this series.

Three hello worlds

#include <stdio.h>

int main(void)
{
    puts("hello from C");
    return 0;
}

fn main() {
    println!("hello from Rust");
}

gcc -O2 hello.c -o hello_c_dyn
gcc -O2 -static hello.c -o hello_c_static
rustc +1.98.0 -O hello.rs -o hello_rs

The machine is a laptop with an Intel Core i7-1260P, Linux 6.6, glibc 2.43, gcc 15.2.1 and rustc 1.98.0. Already the files are very different:

$ wc -c (file size in bytes)
hello_c_dyn          15896
hello_c_static      778440
hello_rs           4507728
hello_rs_nomain    4480600

The fourth file comes later. The static program carries its own copy of the parts of glibc it uses, so it is 49 times bigger. The Rust file is big for another reason, and the next part opens these files.

The first instruction is not mine

strace can print a stack for every system call with -k. For execve this stack shows where the new program is when the call returns, so the first instruction that runs in the process:

$ strace -k ./hello_c_static: the first frame after execve
 > hello_c_static(_start+0x0) [0x2de0]

$ strace -k ./hello_c_dyn: the first frame after execve
 > /usr/lib/ld-linux-x86-64.so.2() [0x1ec40]

For the static program, the first instruction is _start, inside my file. For the dynamic one, it is at offset 0x1ec40 of another file, ld-linux-x86-64.so.2. This is the dynamic loader, and 0x1ec40 is exactly the entry point written in its own header:

$ readelf -h /usr/lib/ld-linux-x86-64.so.2 | grep Entry
  Entry point address:               0x1ec40

How did the kernel know about this file? My program asked for it. The dynamic binary has one line in its program headers that the static binary does not have:

$ readelf -lW hello_c_dyn | grep -E 'Entry point|interpreter'
Entry point 0x1060
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

To see what the kernel prepared, I stopped the dynamic program in gdb at its first instruction (starti) and printed its memory map. gdb turns off address randomization, and this is why the addresses look so regular. I shortened the path of my program in this output and in the others:

Program stopped.
0x00007ffff7fe3c40 in ?? () from /lib64/ld-linux-x86-64.so.2
555555554000-555555555000 r--p 00000000 103:02 1889869                   hello_c_dyn
555555555000-555555556000 r-xp 00001000 103:02 1889869                   hello_c_dyn
555555556000-555555557000 r--p 00002000 103:02 1889869                   hello_c_dyn
555555557000-555555559000 rw-p 00002000 103:02 1889869                   hello_c_dyn
7ffff7fbf000-7ffff7fc3000 r--p 00000000 00:00 0                          [vvar]
7ffff7fc3000-7ffff7fc5000 r-xp 00000000 00:00 0                          [vdso]
7ffff7fc5000-7ffff7fc6000 r--p 00000000 103:02 11020445                  /usr/lib/ld-linux-x86-64.so.2
7ffff7fc6000-7ffff7ff0000 r-xp 00001000 103:02 11020445                  /usr/lib/ld-linux-x86-64.so.2
7ffff7ff0000-7ffff7ffb000 r--p 0002b000 103:02 11020445                  /usr/lib/ld-linux-x86-64.so.2
7ffff7ffb000-7ffff7ffe000 rw-p 00036000 103:02 11020445                  /usr/lib/ld-linux-x86-64.so.2
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0 
7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

The stop address is 0x7ffff7fe3c40, that is the start of the loader in memory (0x7ffff7fc5000) plus 0x1ec40. At this moment not one instruction of user code ran yet, and the process already has my program in four pieces, the loader in four pieces (plus one anonymous page, the end of the loader's zeroed data), a stack, and small areas from the kernel like [vvar] and [vdso]. There is no libc and no heap.

So everything in this map was done by the kernel inside execve. The function that does it is load_elf_binary in fs/binfmt_elf.c. It reads the ELF header and the program headers of my file, finds the interpreter line, opens the interpreter, and maps the loadable segments of both files. Then it builds the new stack with the arguments, the environment and a list of values for the loader called the auxiliary vector, and it sets the instruction pointer. If there is an interpreter, the process starts in the interpreter. If not, like for my static program, it starts at the entry point of the program itself.

A later part of the series reads this initial stack value by value. For a dynamic program, the kernel does not start my program. It starts the loader and gives it my program.

The loader: 27 calls to find and map libc

Here is the start of the trace of the dynamic C program, the part before libc is loaded:

execve("./hello_c_dyn", ["./hello_c_dyn"], 0x7ffe1c7f8900 /* 115 vars */) = 0
brk(NULL)                               = 0x562172c7f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=185747, ...}) = 0
mmap(NULL, 185747, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f98545b8000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0y\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 840, 64) = 840
fstat(3, {st_mode=S_IFREG|0755, st_size=2010168, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f98545b6000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 840, 64) = 840
mmap(NULL, 2034544, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f98543c5000
mmap(0x7f98543e9000, 1511424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24000) = 0x7f98543e9000
mmap(0x7f985455a000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7f985455a000
mmap(0x7f98545a8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e2000) = 0x7f98545a8000
mmap(0x7f98545ae000, 31600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f98545ae000
close(3)                                = 0

Let me go through it slowly.

  • brk(NULL) only asks the kernel where the heap ends now. It changes nothing.
  • access("/etc/ld.so.preload") fails with ENOENT. The loader checks if the system has a list of libraries to load into every program. This machine has none.
  • Then it maps all 185,747 bytes of /etc/ld.so.cache, a precompiled list of where the libraries are, so it does not have to search directories. A munmap gives this memory back later.
  • It opens libc.so.6 and reads the first 832 bytes, which contain the ELF header. The pread64 of 840 bytes at offset 64 reads the program headers: libc has 15 of them, 56 bytes each.
  • The anonymous mmap of 8 KiB is for the loader's own small allocator, and the second pread64 reads the same program headers again.
  • The first big mmap reserves 2,034,544 bytes, the full size of libc in memory. The next four calls place the pieces of libc inside this reservation with MAP_FIXED: the code with read and execute permission, read-only data, writable data, and an anonymous zeroed area for the variables that start at zero.

This is the work the kernel did for my program, now done by the loader for libc with normal system calls.

After this, the loader prepares the thread:

mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f98543c2000
arch_prctl(ARCH_SET_FS, 0x7f98543c2740) = 0
set_tid_address(0x7f98543c2d68)         = 691145
set_robust_list(0x7f98543c2a20, 24)     = 0
rseq(0x7f98543c26a0, 0x20, 0, 0x53053053) = 0
mprotect(0x7f98545a8000, 16384, PROT_READ) = 0
mprotect(0x562137332000, 4096, PROT_READ) = 0
mprotect(0x7f985461c000, 8192, PROT_READ) = 0

The 12 KiB mapping is the memory of the main thread for its own data: the thread-local variables (like errno) and the structure that glibc keeps for each thread. arch_prctl(ARCH_SET_FS) points the fs register at it. Look at the addresses: the value of fs and the next three all fall inside this same 12 KiB area. set_tid_address tells the kernel where to write zero when this thread ends, set_robust_list registers a list of locks for the case where the thread dies while holding one, and rseq registers an area for restartable sequences. All three only give an address to the kernel.

Then three mprotect calls make some memory read-only: one area in libc, one page of my program, one area of the loader. The loader wrote addresses into these pages while it connected the program with libc, and now they become read-only so that nobody can change them later. Two later parts explain what is written there and why.

I did not want to guess which code made each call, so I made a small script that reads the strace -k stacks and prints the file of the innermost frame for every call. The end of its table for the dynamic C program:

 27  mprotect          ld-linux-x86-64.so.2      -
 28  prlimit64         libc.so.6                 getrlimit < __libc_early_init
 29  getrandom         libc.so.6                 -
 30  munmap            ld-linux-x86-64.so.2      -
 31  fstat             libc.so.6                 __fstat64 < _IO_file_doallocate
 32  brk               libc.so.6                 brk < sbrk
 33  brk               libc.so.6                 brk < sbrk
 34  write             libc.so.6                 __write < _IO_file_write

system calls after execve and before the write: 32
  made from ld-linux-x86-64.so.2: 27
  made from libc.so.6: 5

The dash means the frames have no name: the loader on this system has no full symbol table, only the few names that it exports, like _dl_catch_exception. 27 of the 32 calls are the loader, only 5 come from libc.

libc, main, and a surprise at the end

The loader runs the initialization of libc before it jumps to my program. Call 28 is __libc_early_init asking for the stack size limit (8 MiB here). Call 29 asks the kernel for 8 random bytes. In the dynamic libc this function has no exported name, but the same call in the static binary has one: __ptmalloc_init, the setup of malloc.

Calls 31 to 33 come from puts. The first time something is written to stdout, glibc allocates a buffer for it. fstat tells glibc what kind of file stdout is, and the two brk calls are malloc creating its heap for this buffer, 136 KiB at once.

Then the write. I expected it inside main, but the full stack of this call says something different:

write(1, "hello from C\n", 13)          = 13
  libc.so.6                ?
  libc.so.6                ?
  libc.so.6                __write
  libc.so.6                _IO_file_write
  libc.so.6                ?
  libc.so.6                _IO_do_write
  libc.so.6                _IO_flush_all
  libc.so.6                ?
  libc.so.6                ?
  libc.so.6                exit
  libc.so.6                ?
  libc.so.6                __libc_start_main
  hello_c_dyn              _start

main is not there. The line was written by exit, after main returned. Why is it like this? In my test stdout is a file, and glibc buffers output to a file in blocks: puts only copied the line in the buffer, and the buffer was flushed when the process exited. On a terminal stdout is line buffered, so the same puts would write at once. This difference has its own part at the end of the series.

The static binary does the same work by itself

A static program has no loader, but it still needs a thread pointer, a heap and a buffer for stdout. Here is the full table for hello_c_static, with names from its own symbol table:

  #  system call       innermost frame in        named functions on the stack, innermost first
  1  execve            hello_c_static            _start
  2  brk               hello_c_static            _dl_early_allocate < __libc_setup_tls
  3  brk               hello_c_static            _dl_early_allocate < __libc_setup_tls
  4  arch_prctl        hello_c_static            __libc_setup_tls < __libc_start_main_impl
  5  set_tid_address   hello_c_static            __tls_init_tp < __libc_setup_tls
  6  set_robust_list   hello_c_static            __tls_init_tp < __libc_setup_tls
  7  rseq              hello_c_static            __tls_init_tp < __libc_setup_tls
  8  prlimit64         hello_c_static            getrlimit < __libc_start_main_impl
  9  getrandom         hello_c_static            __ptmalloc_init < __libc_start_main_impl
 10  readlinkat        hello_c_static            _dl_get_origin < _dl_non_dynamic_init
 11  brk               hello_c_static            __brk < sysmalloc
 12  brk               hello_c_static            __brk < sysmalloc
 13  brk               hello_c_static            __brk < sysmalloc
 14  mprotect          hello_c_static            __mprotect < __libc_init_first
 15  fstat             hello_c_static            fstat64 < _IO_doallocbuf
 16  write             hello_c_static            __internal_syscall_cancel < __syscall_cancel

It is the same story with fewer steps. The thread area comes from the heap with brk instead of mmap, then arch_prctl and the three registrations, then libc's early setup and malloc. No file is opened, nothing is searched.

One line surprised me: readlinkat of /proc/self/exe, from a function called _dl_get_origin. The static glibc still contains a small part of the dynamic loader, because a static program can call dlopen, and this part asks where the executable lives.

What Rust adds before main

The Rust program loads two libraries, libgcc_s.so.1 and libc.so.6. libgcc_s gives the unwinder that the standard library uses for panics (the binary imports _Unwind_RaiseException, _Unwind_Resume and others). So the loader part is longer, 36 calls instead of 27. After the loader and libc's early setup, the Rust table continues like this:

 40  poll              libc.so.6                 poll < std::sys::pal::unix::init::sanitize_standard_fds
 41  rt_sigaction      libc.so.6                 __libc_sigaction < ssignal
 42  brk               libc.so.6                 brk < sbrk
 43  brk               libc.so.6                 brk < sbrk
 44  openat            libc.so.6                 __open64_nocancel < _IO_file_open
 45  prlimit64         libc.so.6                 getrlimit < pthread_getattr_np
 ...
 53  sched_getaffinity libc.so.6                 pthread_getaffinity_np < pthread_getattr_np
 54  rt_sigaction      libc.so.6                 __libc_sigaction < std::sys::pal::unix::stack_overflow::imp::init
 55  sigaltstack       libc.so.6                 sigaltstack < std::sys::pal::unix::stack_overflow::imp::make_handler
 56  mmap              libc.so.6                 __mmap < std::sys::pal::unix::stack_overflow::imp::get_stack
 57  mprotect          libc.so.6                 __mprotect < std::sys::pal::unix::stack_overflow::imp::get_stack
 58  sigaltstack       libc.so.6                 sigaltstack < std::sys::pal::unix::stack_overflow::imp::make_handler
 59  gettid            libc.so.6                 gettid < std::sys::thread::unix::current_os_id::gettid
 60  rt_sigaction      libc.so.6                 __libc_sigaction < std::sys::pal::unix::stack_overflow::imp::init
 61  rt_sigaction      libc.so.6                 __libc_sigaction < std::sys::pal::unix::stack_overflow::imp::init
 62  rt_sigaction      libc.so.6                 __libc_sigaction < std::sys::pal::unix::stack_overflow::imp::init
 63  write             libc.so.6                 __write < <std::sys::fd::unix::FileDesc>::write

Here the Rust symbol table helps, because the names point into the standard library source. I read the 1.98.0 source of std::rt and std::sys::pal::unix. The C main of a Rust program is generated by rustc and calls lang_start, which calls an init function, then my main, then a cleanup. On Linux, init does three things you can see in the table:

  1. poll on file descriptors 0, 1 and 2, with a timeout of zero. If one of them is closed, std opens /dev/null on it, so that a file opened later can never become "stdout" by accident.
  2. One rt_sigaction sets SIGPIPE to "ignore". Then a write to a closed pipe returns an error that Rust code can handle, and the signal does not kill the process.
  3. The rest prepares the detection of stack overflow. std asks glibc for the limits of the main thread's stack with pthread_getattr_np, and glibc finds them by reading /proc/self/maps with stdio: that is calls 44 to 53, the openat, the reads, the close, etc. Then std maps a small alternate stack for signal handlers, protects its first page, and installs handlers for SIGSEGV and SIGBUS. Because of this, a Rust stack overflow prints a message instead of a bare segmentation fault.

After main, the trace has two more calls than C, sigaltstack and munmap: the cleanup removes the alternate stack before exit_group. And the write is in a normal place this time, inside the print code: in the 1.98 source, Rust's stdout is always a LineWriter, also when it is a file.

To measure this part separately I made a fourth program, hello_rs_nomain. It uses #![no_main] and defines the C main itself, so lang_start never runs. Same libraries, same loader work, and only 40 calls before the write, 4 of them from libc.

What one start costs

To measure a start I made a small launcher in C. It starts a program with posix_spawn, waits for it with wait4, and measures the time between the two. Each program runs 50 times to warm up and then 1000 times, with stdout sent to /dev/null, pinned to one performance core with taskset -c 2. wait4 also gives the minor page faults of each child. The laptop was not idle: the load average was about 3, and other experiments ran the same evening, some of them pinned to this same core. Each program measured alone:

program          starts    median       p10       p90       min       max  faults
hello_c_dyn        1000    280 us    265 us    396 us    252 us   1345 us      63
hello_c_static     1000    111 us    108 us    136 us    106 us    893 us      31
hello_rs           1000    400 us    386 us    517 us    371 us   1876 us      81
hello_rs_nomain    1000    332 us    321 us    441 us    307 us   1422 us      75

Here is the interesting part. The static C program starts in 111 microseconds, and this includes creating the process, execve, running, exiting and waking up the parent. The dynamic one needs 280. So on my machine dynamic linking adds about 170 microseconds to a start that costs 111 without it. Only a part of this is the code of the loader (a later part measures it alone). The page faults show where some of it goes: 31 for the static program, 63 for the dynamic one. A page fault here is the kernel filling one page of memory the first time the process touches it, and the dynamic program touches more pages: the loader, libc, and the data that the loader writes.

Rust needs 400 microseconds. Without the std runtime setup it needs 332, so the setup from the previous section costs about 70 microseconds. The remaining difference with C, about 50 microseconds, is the second library, the bigger binary and more work for the loader.

Honestly, I would not take these absolute numbers to another machine. They move between runs. When the launcher starts the four programs in turn, the static one becomes clearly slower (171 instead of 111 microseconds) and the others almost do not change, so there dynamic linking costs only about 110 microseconds. I do not know the reason. A second full run of the fixture, almost three hours later, gave the same numbers within 5%. Without taskset all the medians were about three times higher and much more spread. The order of the four programs stayed the same in every run I made, so this is what I trust. The full results are in p01-startup-time.txt.

Who does what

So when I type ./hello_c_dyn, this is the order:

  1. The kernel, inside execve, maps my file and the loader, builds the stack and jumps to the loader.
  2. The loader maps libc, prepares the main thread, connects my program to libc and makes some pages read-only.
  3. libc initializes itself, still before my code.
  4. In Rust, std checks fds 0 to 2, ignores SIGPIPE and installs its stack overflow handler.
  5. My main runs. In C the output may wait in a buffer until exit.

For a one-line program, steps 1 to 4 are almost the whole life of the process.

Where this series goes

This is the first part of "From source to a running process". Each part follows one step of this path. First the files: what is inside an executable, where variables live, linker errors and relocations. Then the loading: static and dynamic linking, how libraries are found and called, LD_PRELOAD, the code before main and the first stack. Then the running process: its memory map, page faults, fork, malloc and free, system calls, stack overflows and thread-local variables. The last part is about the exit.

All the parts are on the series page, and the topic page Rust under the hood has the other articles about what Rust becomes on the machine.

Sources