rustc allows code to perform compile-time tests against target_env [0],
which is a function of the target triple. There isn't provision in
Nixpkgs for learning target_env, however. That doesn't affect rustc's
evaluation of cfg guards in Rust code - rustc knows perfectly well what
target_env is - but it _does_ affect the env vars passed to a build
script set by buildRustCrate, which is presently hard-coded to gnu, and
also affects any Nix code looking to branch on target_env.
Being able to access target_env is very relevant to, for example, WASI
as (using Rust terminology) wasm32-wasip1 and wasm32-wasip2 differ only
by target_env, with identical target_arch and target_os values. Properly
reflecting target_env may also fix some random musl issues if you're a
bit lucky.
This commit adds a new attr, rust.platform.env, which reflects the
target_env that rustc will set for the target, and wires it up in
buildRustCrate. In isolation, this change mostly only affects build
scripts checking target_env, but crate2nix will greatly benefit from
being able to accurately resolve the dep graph (which can also vary
depending on target_env).
The target triple -> target_env function resists Kolmogorov compression:
it's irregular and, though there are some patterns, there are lots of
special cases. So, I have done the stupidest possible thing and scraped
out all of the targets with non-empty target_env values and dumped that
into an attrset. This attrset will progressively get out of date as
rustc adds new platforms, but updating it should be simple enough - I've
included the generation script as a comment.
There are some other configuration options not being reflected in Nix. I
have left those alone, but, in the future, maybe this can be extended to
just dumping all of them into an attrset and then reflecting them in
rust.platform. It might even make sense to convert the existing code
producing rust.platform to just looking up from an attrset and to 'do
what rustc does'.
I would have liked to have added a test targetting a platform with a
non-empty non-GNU target_env, but all of the yaks were quite hairy.
Fixes https://github.com/NixOS/nixpkgs/issues/436832
[0]: https://doc.rust-lang.org/reference/conditional-compilation.html#r-cfg.target_env
Scope:
- Combination of
- Textual matches of "baseNameOf (toString"
- Redundant toString calls I've found with my latest
"lazy paths" nix branch as they force lazy fetches into
the store. More info and new PR soon.
- Only cases I believe are worthwhile or easily determined
I've determined the validity by
- testing llvmPackages instantiation
- figuring out which types can pass into any particular
toString call - "human fuzzy type checker"
Behavior considerations by type:
- `path`: converted back to a string *without* context
`baseNameOf` does not copy things to the store on its own,
equivalent to its behavior for string inputs
- `null`: converted to `""` -> may be valid input!
ok if "" would not have been acceptable anyway
- `string` itself: passed through identically -> trivial
- `attrset` with `outPath`: same coercion as built into
the `baseNameOf` function -> trivial
- other atomic types: generally not sensible inputs to
`baseNameOf` -> fuzzy but true
With the addition of `remap-path-prefix` to buildRustCrate, we now
can pull in two rustc variants when cross compiling, since the rustc
that is string-interpolated for passing the remap flag is not pulled
from the proper package-set. Depending on the host platform used, this
rustc variant may not even build. In order to only pull in one variant,
and to properly mask the rustc used for building with the
remap-path-prefix flag, rustc is passed explicitly from pkgsBuildHost.
buildRustCrate calls rustc directly, so there is no built-in equivalent
of `cargo clippy`. The capLints doc already hints at the existing
workaround — overriding the `rust` attribute with a wrapper script that
sniffs argv for build_script_build and routes everything else to
clippy-driver. That works because clippy-driver IS rustc internally
(rustc_driver plus lint passes) and produces compatible rlibs and
metadata, but the wrapper is fragile and every downstream reinvents it.
Add a `useClippy` boolean (default false) that swaps the lib/bin/test
compilation to clippy-driver while keeping plain rustc for build.rs —
the same lib-vs-build-script split as extraRustcOptsForBuildRs, since
build scripts are typically auto-generated and clippy noise there is
not actionable. The clippy-driver binary comes from a new `clippy`
parameter, auto-filled from pkgs.clippy by callPackage and overridable
alongside `rust` for toolchain bundles that ship their own.
Implemented by introducing a RUSTC_DRIVER shell variable in
build-crate.nix that lib.sh's build_lib/build_bin consume; the
build_script_build path in configure-crate.nix never sees it.
Note that the default `capLints = "allow"` cap suppresses clippy lints
along with everything else, so `useClippy` is usually paired with
`capLints = "warn"` and `-W clippy::*`/`-D warnings` flags via
extraRustcOpts. Documented in the manual.
Test fixtures cover the three load-bearing properties: a denied clippy
lint fails the build (proving clippy-driver ran), the default capLints
silences clippy lints just like rustc lints, and an rlib emitted by
clippy-driver is link-compatible with a plain-rustc dependent.
buildRustCrate passes `--remap-path-prefix=$NIX_BUILD_TOP=/` so the
sandbox build directory is not embedded in compiled artifacts. The
resulting source path is `/$sourceRoot/src/...` — which works out to
`/<crateName>-<version>/src/...` only because fetchCrate happens to
unpack to a directory named after the crate.
When `src` is supplied directly — `lib.fileset.toSource` (always
`source`), `lib.cleanSource ./.`, a flake's `self`, or any
`<hash>-source` store path — stdenv unpacks to `$NIX_BUILD_TOP/source/`,
so every such crate ends up with `/source/src/lib.rs`. The crate
identity is gone from panic backtraces, `file!()` expansions, debuginfo,
and llvm-cov coverage maps. Workspace builds via crate2nix that source
each member with a fileset see all members collapse to the same prefix,
making per-crate coverage and backtrace attribution impossible without a
per-crate remap override.
Add a second, more specific remap of `$NIX_BUILD_TOP/$sourceRoot` to
`/<crateName>-<version>`. rustc applies `--remap-path-prefix` flags
last-match-wins, so this prefix wins for everything under the source
root — which in a buildRustCrate build is effectively everything that
would otherwise embed a sandbox path, including `OUT_DIR` (placed at
`$sourceRoot/target/build/` by configure-crate.nix). The broader
`$NIX_BUILD_TOP=/` remap stays as a fallback for any path outside
`$sourceRoot`.
This is safe with respect to the original remap's purpose:
- Reproducibility: still maps the per-build sandbox path to a fixed,
build-independent string. `crateName` and `version` are already
derivation inputs, so the result is deterministic.
- Closure size: still no store paths in the embedded strings.
- fetchCrate sources: `$sourceRoot` is `<crateName>-<version>` there,
so the new remap rewrites `/<crateName>-<version>` to itself — the
output is byte-for-byte identical. Only custom-`src` crates change,
and they change from a degenerate `/source/...` to the same shape
fetchCrate crates already had.
- The remap only affects path *strings* embedded in output (DWARF
`DW_AT_comp_dir`/`DW_AT_name`, `file!()`, `Location::file()`, LLVM
coverage filename tables). It does not change which files are read
or compiled.
Users who depend on the old `/source/...` prefix can restore it via
`extraRustcOpts`, which is appended after these flags and therefore
wins under last-match-wins.
Cargo links [dev-dependencies] only when building test or bench
targets. buildRustCrate models normal deps (`dependencies`) and
build-script deps (`buildDependencies`) but has no equivalent: callers
that set `buildTests = true` for a crate with dev-deps must compute the
dev-dep set themselves and splice it into `dependencies` via .override.
Add a `devDependencies` parameter that is appended to the linked
dependency list only when `buildTests = true`. The append happens at the
single point where `dependencies` is normalized, so it threads through
to the --extern flags, the deps directory in configurePhase, the
metadata hash, and propagated build inputs without further changes.
When `buildTests = false` the parameter is dropped, so existing lib/bin
derivations are byte-identical to before — no rebuilds for users that
don't pass it.
Includes a test case in build-rust-crate/test that fails without the
change (`can't find crate for dev_dep`).
Proc-macro crates are host dylibs that rustc dlopen()s. Instrumentation
flags passed via extraRustcOpts (e.g. -Zsanitizer=address,
-Cinstrument-coverage) leave unresolved runtime symbols in those dylibs
and break the build. Cargo avoids this by not applying RUSTFLAGS to host
artifacts; buildRustCrate already has extraRustcOptsForBuildRs for build
scripts, so add the analogous knob for proc-macros.
Defaults to null, which falls back to extraRustcOpts so existing callers
are unchanged. Set to [] to opt proc-macros out when applying
sanitizer/coverage flags tree-wide via crateOverrides.
Add completePropagatedBuildInputs that collects propagatedBuildInputs
from a crate and all its transitive Rust dependencies, analogous to how
completeDeps chains .rlib paths. The collected inputs are appended to
buildInputs so native library deps (e.g. boost) declared on a library
crate automatically propagate to binary crates that depend on it,
without requiring repetition in every downstream crate override.
The Makefile generated for parallel binary builds used the binary names
directly as make targets. This caused make to silently skip a build
whenever a file or directory with the same name existed in the crate
source root — e.g. a crate with a `cli` binary and a `cli/` directory
would "succeed" without producing the binary.
Using the binary name as a target also risked collision with the `all`
meta-target and with make metacharacters (`$`, `:`, `#`) in user-supplied
crateName or bin.path values.
Switch to synthetic numeric targets (b0, b1, …) declared .PHONY, and
escape `$` in the recipe arguments.
Collect binary targets into a bash associative array (name→path), then
generate a Makefile on the fly and pipe it to `make -j`. Make handles
parallel execution and the jobserver protocol natively, so rustc
invocations share the same token pool via MAKEFLAGS.
Also fix build_bin error propagation: add `|| return 1` to the rustc
invocation so failures are not silently swallowed when the crate name
has no hyphens and the mv rename is skipped.
Cargo sets `CARGO_BIN_EXE_<name>` when compiling integration tests so
they can locate the crate's own binaries via `env!()` and exec them
as subprocesses. buildRustCrate previously compiled bin targets as
test harnesses only (via `--test`) when buildTests=true, so the real
binary never existed and the env var was never set.
When buildTests is set, now first build the real (non-test) binaries
to `target/cargo-bin-exe/`, install them to `$out/bin/`, and expose
`CARGO_BIN_EXE_<name>` to subsequent rustc invocations via an `env`
prefix. Using `env` rather than bash `export` is necessary because
hyphenated binary names (`CARGO_BIN_EXE_my-crate`) produce env var
names that bash rejects as invalid identifiers; `env` sets them via
execve's envp array directly and has no such restriction.
The env array is populated by iterating `target/cargo-bin-exe/` at
build time, so it covers all binary-target shapes: explicit `crateBin`
entries, auto-detected `src/main.rs`, and auto-detected `src/bin/*.rs`.
`build_bin` in lib.sh gains an optional third argument for the output
directory so the real binary and the `--test` harness can coexist
without colliding in `target/bin/`. The lib-test build is reordered to
after the real-bin build so its compilation also sees the env vars.
Cargo's `[lints]` table is a cargo-only feature: cargo translates the
entries into `-A`/`-W`/`-D`/`-F` rustc flags before invoking the
compiler. Since buildRustCrate calls rustc directly, these lints were
silently ignored, forcing users to duplicate them as raw flags in
`extraRustcOpts`.
Accept a `lints` attr with the same shape as the Cargo.toml section
(tool → lint name → level string or `{ level, priority }` attrset),
translate it to rustc flags in Nix, and append to the rustc command
line. Entries are sorted by ascending priority so lower-priority lint
groups are emitted first and can be overridden by more specific lints,
matching cargo's behaviour.
The `capLints` default changes from `"allow"` to `null`, resolved to
`"allow"` when `lints` is empty (the usual case for third-party
dependencies) and `"forbid"` when `lints` is set — otherwise a
`deny`/`forbid` lint would be silently capped and the table would be a
no-op. Explicit `capLints` still overrides. This is the same model
cargo uses: your own crate's `[lints]` always apply; only
dependencies get `--cap-lints allow`.
Generators like crate2nix can populate `lints` directly from the
manifest at generation time.
build_bin_test_file derives the test binary name by replacing `/` with
`_`, stripping the `tests_` prefix and `.rs` suffix. For the
subdirectory integration-test layout `tests/<dir>/main.rs`, this
produces `<dir>_main`, but cargo names that binary `<dir>` per its
auto-discovery rules. Tools that expect the cargo convention
(cargo-nextest binaries-metadata, IDE test runners) cannot find the
binaries.
Strip the trailing `_main` when the source file is `*/main.rs`. The
guard is needed so a flat-style `tests/<name>_main.rs` keeps its
suffix.
Add `expectedTestBinaries` to the test harness so binary naming can be
asserted, and cover both the subdir case (suffix stripped) and the
flat case with a literal `_main` in the filename (suffix kept). Also
fix a typo in `removeAttrs` that leaked `expectedTestOutputs` into the
crate args.
lib.sh previously hardcoded `--cap-lints allow` in both build_lib and
build_bin. Since rustc only honours the first `--cap-lints` it sees,
appending a different level via `extraRustcOpts` had no effect, making
it impossible to run clippy (or any lint-based tooling) through
buildRustCrate without wrapper scripts that strip the argument.
This adds a `capLints` parameter (default `"allow"`, preserving current
behaviour) that can be set per-crate or via `.override`:
(myCrate { }).override { capLints = "warn"; }
When the rust-src component is installed (common with rust-overlay
toolchains via rust-toolchain.toml), rustc unvirtualises libstd source
paths. Panic locations from monomorphised generic std code
(BTreeMap, VecDeque, sync primitives, etc.) then embed the toolchain
store path in .rodata, pulling the entire multi-GB toolchain into the
runtime closure.
This is not an RPATH issue — stdenv's patchelf --shrink-rpath already
cleans that. The reference is a core::panic::Location string.
Remap the rustc store path to /rustc so panic messages stay readable
but no longer create a store-path dependency. This is a no-op for
toolchains without rust-src (the prefix simply never matches).
this creates some eval errors that will be fixed in the next commit
done with the following script:
```fish
\#!/usr/bin/env fish
set packagesjson (nix eval --impure --json --expr '
let
lib = import ./lib;
in
import pkgs/servers/x11/xorg/default.nix (lib.mapAttrs (
name: _:
if name == "lib" then
lib
else if name == "config" then
{ allowAliases = false; }
else
name
) (__functionArgs (import pkgs/servers/x11/xorg/default.nix))) { }
' | jq)
set one (grep '^ [A-Za-z0-9_-]*$' pkgs/servers/x11/xorg/default.nix | string trim | string replace -r '$' Z | sort | string sub -e -1)
set two (grep '^ [A-Za-z0-9_-]* = [A-Za-z0-9_-]*;$' pkgs/servers/x11/xorg/default.nix | cut -d= -f1 | string trim | string replace -r '$' Z | sort | string sub -e -1)
for arg in $one $two
set oname $arg
set nname (echo $packagesjson | jq -r .$oname)
if test $nname = null
echo (set_color red)warn:(set_color normal) unknown package xorg.$oname >&2
continue
end
echo $oname "->" $nname
# replace basic xorg.$name references
for file in (rg -F "xorg.$oname" --files-with-matches pkgs)
# special cases
sd -F "$oname = xorg.$oname;" "$nname = $nname;" $file
# replace
sd -F "xorg.$oname" "$nname" $file
# fixup function arguments
# prevent duplicate function args
if grep -E " ($oname|$nname),\$" $file >/dev/null
continue
end
if grep 'xorg\..' $file >/dev/null # case1: there is more so we can't just remove the function arg
if grep ' xorg,$' $file >/dev/null
sd ' xorg,$' " xorg,
$nname," $file
else if grep ' xorg ? .*,$' $file >/dev/null
sd 'xorg( ? .*),$' "xorg\$1,
$nname," $file
else
sd -F 'xorg,' "$nname,
xorg," $file
end
else # case there is no more xorg..* so we can just replace the function arg
sd 'xorg(| ? .*),.*$' "$nname," $file
end
end
end
nix fmt
```
These are currently broken because buildRustCrate will try to mv from
an underscored name to a hyphenated name, but will only try to move an
unsuffixed name; rustc generates wasm binaries with a .wasm suffix.
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](8616af08d9/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
After final improvements to the official formatter implementation,
this commit now performs the first treewide reformat of Nix files using it.
This is part of the implementation of RFC 166.
Only "inactive" files are reformatted, meaning only files that
aren't being touched by any PR with activity in the past 2 months.
This is to avoid conflicts for PRs that might soon be merged.
Later we can do a full treewide reformat to get the rest,
which should not cause as many conflicts.
A CI check has already been running for some time to ensure that new and
already-formatted files are formatted, so the files being reformatted here
should also stay formatted.
This commit was automatically created and can be verified using
nix-build a08b3a4d19.tar.gz \
--argstr baseRev b32a094368
result/bin/apply-formatting $NIXPKGS_PATH
In preparation for the deprecation of `stdenv.isX`.
These shorthands are not conducive to cross-compilation because they
hide the platforms.
Darwin might get cross-compilation for which the continued usage of `stdenv.isDarwin` will get in the way
One example of why this is bad and especially affects compiler packages
https://www.github.com/NixOS/nixpkgs/pull/343059
There are too many files to go through manually but a treewide should
get users thinking when they see a `hostPlatform.isX` in a place where it
doesn't make sense.
```
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv.is" "stdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv'.is" "stdenv'.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "clangStdenv.is" "clangStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "gccStdenv.is" "gccStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenvNoCC.is" "stdenvNoCC.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "inherit (stdenv) is" "inherit (stdenv.hostPlatform) is"
fd --type f "\.nix" | xargs sd --fixed-strings "buildStdenv.is" "buildStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "effectiveStdenv.is" "effectiveStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "originalStdenv.is" "originalStdenv.hostPlatform.is"
```
I broke the usage of buildRustCrate with a toolchain from rust-overlay
when I added support for wasm32-unknown-unknown, this change adds
additional conditionals to restore the usage.
The toolchain can now be overriden either through
```
buildRustCrate {
rust = toolchain;
cargo = toolchain
}
```
or
```
buildRustCrate.override {
rustc = toolchain;
cargo = toolchain;
}
```
They should now be consistent with each other.
