A debuginfod support must be able to map a build-id to
- debug symbols
- the original elf file for which the debug symbols where separated
- the corresponding source files
Currently, hydra provides an index from build-id to the nar of the debug
output containing the debug symbols.
Add symlinks in these outputs so that we can recover the store path of
the source and original elf file. We can then fetch them by the normal
binary cache protocol.
About source files: to minimize storage demands, in the ideal case,
software would be built from the source store path $src and the
debuginfod server would just have to serve source files from this store
path. In practice, source files are sometimes patched as part of the
build. This commit stores the modified files in the debug output is a so
called source overlay so that the debuginfod serve can serve the patched
content of the file.
The checksum was chosen as follows (where big is 4GB of zeros):
$ hyperfine -L s sysv,bsd,crc,sha1,sha224,sha256,sha384,sha512,blake2b,sm3 'cksum -a {s} big'
Benchmark 1: cksum -a sysv big
Time (mean ± σ): 854.5 ms ± 270.5 ms [User: 245.3 ms, System: 601.8 ms]
Range (min … max): 760.5 ms … 1623.8 ms 10 runs
Warning: The first benchmarking run for this command was significantly slower than the rest (1.624 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
Benchmark 2: cksum -a bsd big
Time (mean ± σ): 5.838 s ± 0.045 s [User: 5.118 s, System: 0.693 s]
Range (min … max): 5.767 s … 5.897 s 10 runs
Benchmark 3: cksum -a crc big
Time (mean ± σ): 829.9 ms ± 28.6 ms [User: 274.5 ms, System: 551.0 ms]
Range (min … max): 803.2 ms … 904.8 ms 10 runs
Benchmark 4: cksum -a sha1 big
Time (mean ± σ): 2.553 s ± 0.010 s [User: 1.912 s, System: 0.631 s]
Range (min … max): 2.543 s … 2.575 s 10 runs
Benchmark 5: cksum -a sha224 big
Time (mean ± σ): 2.716 s ± 0.018 s [User: 2.054 s, System: 0.645 s]
Range (min … max): 2.695 s … 2.743 s 10 runs
Benchmark 6: cksum -a sha256 big
Time (mean ± σ): 2.751 s ± 0.029 s [User: 2.057 s, System: 0.674 s]
Range (min … max): 2.712 s … 2.812 s 10 runs
Benchmark 7: cksum -a sha384 big
Time (mean ± σ): 5.600 s ± 0.049 s [User: 4.820 s, System: 0.753 s]
Range (min … max): 5.515 s … 5.683 s 10 runs
Benchmark 8: cksum -a sha512 big
Time (mean ± σ): 5.543 s ± 0.021 s [User: 4.751 s, System: 0.768 s]
Range (min … max): 5.523 s … 5.579 s 10 runs
Benchmark 9: cksum -a blake2b big
Time (mean ± σ): 5.091 s ± 0.025 s [User: 4.306 s, System: 0.764 s]
Range (min … max): 5.048 s … 5.125 s 10 runs
Benchmark 10: cksum -a sm3 big
Time (mean ± σ): 14.220 s ± 0.120 s [User: 13.376 s, System: 0.783 s]
Range (min … max): 14.077 s … 14.497 s 10 runs
Summary
cksum -a crc big ran
1.03 ± 0.33 times faster than cksum -a sysv big
3.08 ± 0.11 times faster than cksum -a sha1 big
3.27 ± 0.11 times faster than cksum -a sha224 big
3.31 ± 0.12 times faster than cksum -a sha256 big
6.13 ± 0.21 times faster than cksum -a blake2b big
6.68 ± 0.23 times faster than cksum -a sha512 big
6.75 ± 0.24 times faster than cksum -a sha384 big
7.03 ± 0.25 times faster than cksum -a bsd big
17.13 ± 0.61 times faster than cksum -a sm3 big
unfortunately, crc (and sysv) are not supported by --check, so they are
disqualified. sha1 sha224 and sha256 are sensibly as fast as one
another, so let's use a non broken one, even though cryptographic
qualities are not needed here.
There's no point generating debug info if the compiler immediately
strips it before we get a chance to do anything with it.
This is especially important since Cargo 1.77, which asks rustc to
strip by default.
Setting RUSTFLAGS causes Cargo to ignore other ways of configuring
flags, including the target-specific RUSTFLAGS options. This broke
pkgsCross.musl64.crosvm, and was surprising to users.
Fixes: https://github.com/NixOS/nixpkgs/pull/261727
`$OBJCOPY` is not available in bootstrap tools
`stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.bash.stdenv.cc.bintools`
If multiple files with the same build id were found, we silently randomly
overwrote one with the other.
Change the order to make the output deterministic, and emit a
warning when overwriting.
Currently, separate-debug-info adds a debug output, and the build fail when it is
not created. the output is only created when at least one elf file is
stripped.
As a result, adding separateDebugInfo = true on a lib will break the
static build (unless the lib also contains an executable). In order to
not have to remember to add an exception every time, let's just create
the debug output unconditionally.
Before the change separate-debug-info.sh did the stripping itself.
This scheme has a few problems:
1. Stripping happens only on ELF files. *.a and *.o files are skipped.
Derivations have to do it manually. Usually incorrectly
as they don't run $RANLIB (true for `glibc` and `musl`).
2. Stripping happens on all paths. Ideally only `stripDebugList` paths
should be considered.
3. Host strip is called on Target files.
This change offloads stripping logic to strip hook. This strips more
files for `glibc` and `musl`. Now we can remove most $STRIP calls
from individual derivations.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
By default, Cargo will only enable line tables. -g enables full debug
info. The RUSTFLAGS environment variable is examined by Cargo,
similar to how the NIX_*FLAGS* variables are examined by our compiler
wrappers.
This reverts commit 848091a52b, reversing
changes made to ab0e692ac7.
It caused issues with elfutils tests,
probably through over-stripping of glibc parts.
According to https://stackoverflow.com/q/46197810/115030,
--only-keep-debug preserves all the information stripped by
--strip-unneeded. This reduces the size of the webkitgtk output by 22%
(123 MB → 96 MB).
Inspired by #159612.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
find fails when called with an inexistent search path.
That situation may arise when the output is created after by a postFixup hook.
vcunat amended the PR by clarifying one more `return` to `return 0`.
The importance of glibc makes it worthwhile to provide debug
symbols. However, this revealed an issue with separateDebugInfo: it
was indiscriminately adding --build-id to all ld invocations, while in
fact it should only do that for final links. Glibc also uses non-final
("relocatable") links, leading to subsequent failure to apply a build
ID ("Cannot create .note.gnu.build-id section, --build-id
ignored"). So now ld-wrapper.sh only passes --build-id for final
links.
For instance, a binary like libfoo.so will cause a symlink
lib/debug/libfoo.so.debug -> .build-id/<build-ID>.debug to be
created. This is primarily useful for use with eu-addr2line, if you
know the name of a binary and the relative address, but not the build
ID.
You can now pass
separateDebugInfo = true;
to mkDerivation. This causes debug info to be separated from ELF
binaries and stored in the "debug" output. The advantage is that it
enables installing lean binaries, while still having the ability to
make sense of core dumps, etc.
