Keep the debug symbols for the binary you actually shipped

Article · Through the layers

A production crash leaves a list of addresses. The release binary was stripped, the CI artifact expired, and someone suggests rebuilding the same commit to recover the symbols.

That rebuild may not produce the same addresses. Toolchain, dependencies, flags, generated code, and linking can all affect the result.

I would preserve the debugging material when producing the release. It is much easier than reconstructing a build during an incident.

Keep one identity across the release

Record the source revision, toolchain, target, profile, lockfile identity, and executable identity with the artifact. A platform build ID or binary digest can help match a crash to the exact executable.

The Rust stack-trace article explains how frames connect to source. That connection depends on matching information. A familiar function name from the wrong build can make an investigation confidently wrong.

Cargo's profile documentation describes debug information and stripping. For example, a release profile can request line information:

[profile.release]
debug = "line-tables-only"

This setting alone is not a complete symbol-storage workflow. The placement and extraction of debug data depend on the platform and toolchain. Decide how your release keeps a debuggable artifact and produces the executable it deploys.

Do the extraction or splitting from that release build. Do not compile a second “debug equivalent” and assume its addresses match the production executable.

Verify the path with a deliberate crash

Use a small Rust release fixture that fails at a known source line. Build it through the normal CI path, preserve the relevant artifacts, and resolve its trace using the retained material.

Check the deployed executable identity against the stored record. Add a deliberately mismatched artifact and make sure the procedure detects or clearly rejects the mismatch rather than accepting a plausible result.

The point is to test the retrieval and matching procedure, not only whether debug information was requested in Cargo.toml. If an on-call engineer cannot find the artifact after a deployment, the configuration did not solve the operational problem.

Choose retention based on supported releases

Keep symbols for versions that can still be running, including rollback versions and long-lived customer deployments where relevant. A short default CI expiry may be fine for intermediate builds and inadequate for releases.

Restrict access appropriately. Debugging artifacts can expose source paths and internal structure. Core dumps can contain much more, including credentials and user data, so give them a separate collection and retention policy.

A container image tag is not enough identity when tags can move. Record the immutable image or binary identity that actually reached the environment.

Some traces will still be incomplete

Optimizations, inlining, missing unwind information, and the failure mode itself can limit a trace. A symbol archive does not promise a perfect call stack after every crash.

It does give the investigation the correct material. Keep the build identity, deployment record, and retrieval instructions together so the first incident does not become the first test of that process.

For a service you operate, I would consider recoverable debugging information part of shipping. The feature is not complete when the binary starts. Somebody must also be able to understand where that exact binary failed.

One binary, matching symbols, and a deliberate mismatch

The symbol trial compiles a tiny Rust program with a retained marker function and a controlled panic. It keeps the unstripped output long enough to extract a separate debug file, then strips the shipped copy and attaches a GNU debug link.

The fixture is deliberately specific: Linux on x86-64, a non-PIE executable, optimization level one, full debug information, and a linker build ID. Those flags make the marker address directly usable in this reproduction. They are not a general recommendation to disable position-independent executables in a production service. PIE and shared-library addresses need the correct load-address adjustment.

In the recorded run, the stripped binary and debug file share build ID 77b43b2df9eea39f0d95517b71143d1f7dfb89da. Resolving the emitted marker address against that debug file finds retained_marker at symbols.rs:3. The evidence also retains the stripped binary's SHA-256, exact flags, and Rust toolchain identity.

Then the driver changes the panic text and builds another executable. That executable has a different build ID. The resolver refuses to pair it with the first symbol file. This is the useful failure case: a plausible source line from the wrong release would be worse than a clear mismatch.

The Cargo profile above is one retention choice. The fixture itself uses full debug information to verify the mechanism. Decide how much information your profiler, crash collector, and support process need, and test that exact combination before reducing it.

The trial resolves a known marker from a deliberate failure. It does not recover every optimized frame or prove a production crash collector is configured correctly. Extend the release check with a trace from the actual packaged service, and retain symbols for each release still in use. Rebuilding the same source later is weaker than retaining the matching artifact now.