rustc allows code to perform compile-time tests against target_env [0],
which is a function of the target triple. There isn't provision in
Nixpkgs for learning target_env, however. That doesn't affect rustc's
evaluation of cfg guards in Rust code - rustc knows perfectly well what
target_env is - but it _does_ affect the env vars passed to a build
script set by buildRustCrate, which is presently hard-coded to gnu, and
also affects any Nix code looking to branch on target_env.
Being able to access target_env is very relevant to, for example, WASI
as (using Rust terminology) wasm32-wasip1 and wasm32-wasip2 differ only
by target_env, with identical target_arch and target_os values. Properly
reflecting target_env may also fix some random musl issues if you're a
bit lucky.
This commit adds a new attr, rust.platform.env, which reflects the
target_env that rustc will set for the target, and wires it up in
buildRustCrate. In isolation, this change mostly only affects build
scripts checking target_env, but crate2nix will greatly benefit from
being able to accurately resolve the dep graph (which can also vary
depending on target_env).
The target triple -> target_env function resists Kolmogorov compression:
it's irregular and, though there are some patterns, there are lots of
special cases. So, I have done the stupidest possible thing and scraped
out all of the targets with non-empty target_env values and dumped that
into an attrset. This attrset will progressively get out of date as
rustc adds new platforms, but updating it should be simple enough - I've
included the generation script as a comment.
There are some other configuration options not being reflected in Nix. I
have left those alone, but, in the future, maybe this can be extended to
just dumping all of them into an attrset and then reflecting them in
rust.platform. It might even make sense to convert the existing code
producing rust.platform to just looking up from an attrset and to 'do
what rustc does'.
I would have liked to have added a test targetting a platform with a
non-empty non-GNU target_env, but all of the yaks were quite hairy.
Fixes https://github.com/NixOS/nixpkgs/issues/436832
[0]: https://doc.rust-lang.org/reference/conditional-compilation.html#r-cfg.target_env
Scope:
- Combination of
- Textual matches of "baseNameOf (toString"
- Redundant toString calls I've found with my latest
"lazy paths" nix branch as they force lazy fetches into
the store. More info and new PR soon.
- Only cases I believe are worthwhile or easily determined
I've determined the validity by
- testing llvmPackages instantiation
- figuring out which types can pass into any particular
toString call - "human fuzzy type checker"
Behavior considerations by type:
- `path`: converted back to a string *without* context
`baseNameOf` does not copy things to the store on its own,
equivalent to its behavior for string inputs
- `null`: converted to `""` -> may be valid input!
ok if "" would not have been acceptable anyway
- `string` itself: passed through identically -> trivial
- `attrset` with `outPath`: same coercion as built into
the `baseNameOf` function -> trivial
- other atomic types: generally not sensible inputs to
`baseNameOf` -> fuzzy but true
buildRustCrate calls rustc directly, so there is no built-in equivalent
of `cargo clippy`. The capLints doc already hints at the existing
workaround — overriding the `rust` attribute with a wrapper script that
sniffs argv for build_script_build and routes everything else to
clippy-driver. That works because clippy-driver IS rustc internally
(rustc_driver plus lint passes) and produces compatible rlibs and
metadata, but the wrapper is fragile and every downstream reinvents it.
Add a `useClippy` boolean (default false) that swaps the lib/bin/test
compilation to clippy-driver while keeping plain rustc for build.rs —
the same lib-vs-build-script split as extraRustcOptsForBuildRs, since
build scripts are typically auto-generated and clippy noise there is
not actionable. The clippy-driver binary comes from a new `clippy`
parameter, auto-filled from pkgs.clippy by callPackage and overridable
alongside `rust` for toolchain bundles that ship their own.
Implemented by introducing a RUSTC_DRIVER shell variable in
build-crate.nix that lib.sh's build_lib/build_bin consume; the
build_script_build path in configure-crate.nix never sees it.
Note that the default `capLints = "allow"` cap suppresses clippy lints
along with everything else, so `useClippy` is usually paired with
`capLints = "warn"` and `-W clippy::*`/`-D warnings` flags via
extraRustcOpts. Documented in the manual.
Test fixtures cover the three load-bearing properties: a denied clippy
lint fails the build (proving clippy-driver ran), the default capLints
silences clippy lints just like rustc lints, and an rlib emitted by
clippy-driver is link-compatible with a plain-rustc dependent.
Cargo links [dev-dependencies] only when building test or bench
targets. buildRustCrate models normal deps (`dependencies`) and
build-script deps (`buildDependencies`) but has no equivalent: callers
that set `buildTests = true` for a crate with dev-deps must compute the
dev-dep set themselves and splice it into `dependencies` via .override.
Add a `devDependencies` parameter that is appended to the linked
dependency list only when `buildTests = true`. The append happens at the
single point where `dependencies` is normalized, so it threads through
to the --extern flags, the deps directory in configurePhase, the
metadata hash, and propagated build inputs without further changes.
When `buildTests = false` the parameter is dropped, so existing lib/bin
derivations are byte-identical to before — no rebuilds for users that
don't pass it.
Includes a test case in build-rust-crate/test that fails without the
change (`can't find crate for dev_dep`).
Proc-macro crates are host dylibs that rustc dlopen()s. Instrumentation
flags passed via extraRustcOpts (e.g. -Zsanitizer=address,
-Cinstrument-coverage) leave unresolved runtime symbols in those dylibs
and break the build. Cargo avoids this by not applying RUSTFLAGS to host
artifacts; buildRustCrate already has extraRustcOptsForBuildRs for build
scripts, so add the analogous knob for proc-macros.
Defaults to null, which falls back to extraRustcOpts so existing callers
are unchanged. Set to [] to opt proc-macros out when applying
sanitizer/coverage flags tree-wide via crateOverrides.
Add completePropagatedBuildInputs that collects propagatedBuildInputs
from a crate and all its transitive Rust dependencies, analogous to how
completeDeps chains .rlib paths. The collected inputs are appended to
buildInputs so native library deps (e.g. boost) declared on a library
crate automatically propagate to binary crates that depend on it,
without requiring repetition in every downstream crate override.
Cargo sets `CARGO_BIN_EXE_<name>` when compiling integration tests so
they can locate the crate's own binaries via `env!()` and exec them
as subprocesses. buildRustCrate previously compiled bin targets as
test harnesses only (via `--test`) when buildTests=true, so the real
binary never existed and the env var was never set.
When buildTests is set, now first build the real (non-test) binaries
to `target/cargo-bin-exe/`, install them to `$out/bin/`, and expose
`CARGO_BIN_EXE_<name>` to subsequent rustc invocations via an `env`
prefix. Using `env` rather than bash `export` is necessary because
hyphenated binary names (`CARGO_BIN_EXE_my-crate`) produce env var
names that bash rejects as invalid identifiers; `env` sets them via
execve's envp array directly and has no such restriction.
The env array is populated by iterating `target/cargo-bin-exe/` at
build time, so it covers all binary-target shapes: explicit `crateBin`
entries, auto-detected `src/main.rs`, and auto-detected `src/bin/*.rs`.
`build_bin` in lib.sh gains an optional third argument for the output
directory so the real binary and the `--test` harness can coexist
without colliding in `target/bin/`. The lib-test build is reordered to
after the real-bin build so its compilation also sees the env vars.
Cargo's `[lints]` table is a cargo-only feature: cargo translates the
entries into `-A`/`-W`/`-D`/`-F` rustc flags before invoking the
compiler. Since buildRustCrate calls rustc directly, these lints were
silently ignored, forcing users to duplicate them as raw flags in
`extraRustcOpts`.
Accept a `lints` attr with the same shape as the Cargo.toml section
(tool → lint name → level string or `{ level, priority }` attrset),
translate it to rustc flags in Nix, and append to the rustc command
line. Entries are sorted by ascending priority so lower-priority lint
groups are emitted first and can be overridden by more specific lints,
matching cargo's behaviour.
The `capLints` default changes from `"allow"` to `null`, resolved to
`"allow"` when `lints` is empty (the usual case for third-party
dependencies) and `"forbid"` when `lints` is set — otherwise a
`deny`/`forbid` lint would be silently capped and the table would be a
no-op. Explicit `capLints` still overrides. This is the same model
cargo uses: your own crate's `[lints]` always apply; only
dependencies get `--cap-lints allow`.
Generators like crate2nix can populate `lints` directly from the
manifest at generation time.
build_bin_test_file derives the test binary name by replacing `/` with
`_`, stripping the `tests_` prefix and `.rs` suffix. For the
subdirectory integration-test layout `tests/<dir>/main.rs`, this
produces `<dir>_main`, but cargo names that binary `<dir>` per its
auto-discovery rules. Tools that expect the cargo convention
(cargo-nextest binaries-metadata, IDE test runners) cannot find the
binaries.
Strip the trailing `_main` when the source file is `*/main.rs`. The
guard is needed so a flat-style `tests/<name>_main.rs` keeps its
suffix.
Add `expectedTestBinaries` to the test harness so binary naming can be
asserted, and cover both the subdir case (suffix stripped) and the
flat case with a literal `_main` in the filename (suffix kept). Also
fix a typo in `removeAttrs` that leaked `expectedTestOutputs` into the
crate args.
this creates some eval errors that will be fixed in the next commit
done with the following script:
```fish
\#!/usr/bin/env fish
set packagesjson (nix eval --impure --json --expr '
let
lib = import ./lib;
in
import pkgs/servers/x11/xorg/default.nix (lib.mapAttrs (
name: _:
if name == "lib" then
lib
else if name == "config" then
{ allowAliases = false; }
else
name
) (__functionArgs (import pkgs/servers/x11/xorg/default.nix))) { }
' | jq)
set one (grep '^ [A-Za-z0-9_-]*$' pkgs/servers/x11/xorg/default.nix | string trim | string replace -r '$' Z | sort | string sub -e -1)
set two (grep '^ [A-Za-z0-9_-]* = [A-Za-z0-9_-]*;$' pkgs/servers/x11/xorg/default.nix | cut -d= -f1 | string trim | string replace -r '$' Z | sort | string sub -e -1)
for arg in $one $two
set oname $arg
set nname (echo $packagesjson | jq -r .$oname)
if test $nname = null
echo (set_color red)warn:(set_color normal) unknown package xorg.$oname >&2
continue
end
echo $oname "->" $nname
# replace basic xorg.$name references
for file in (rg -F "xorg.$oname" --files-with-matches pkgs)
# special cases
sd -F "$oname = xorg.$oname;" "$nname = $nname;" $file
# replace
sd -F "xorg.$oname" "$nname" $file
# fixup function arguments
# prevent duplicate function args
if grep -E " ($oname|$nname),\$" $file >/dev/null
continue
end
if grep 'xorg\..' $file >/dev/null # case1: there is more so we can't just remove the function arg
if grep ' xorg,$' $file >/dev/null
sd ' xorg,$' " xorg,
$nname," $file
else if grep ' xorg ? .*,$' $file >/dev/null
sd 'xorg( ? .*),$' "xorg\$1,
$nname," $file
else
sd -F 'xorg,' "$nname,
xorg," $file
end
else # case there is no more xorg..* so we can just replace the function arg
sd 'xorg(| ? .*),.*$' "$nname," $file
end
end
end
nix fmt
```
These are currently broken because buildRustCrate will try to mv from
an underscored name to a hyphenated name, but will only try to move an
unsuffixed name; rustc generates wasm binaries with a .wasm suffix.
After final improvements to the official formatter implementation,
this commit now performs the first treewide reformat of Nix files using it.
This is part of the implementation of RFC 166.
Only "inactive" files are reformatted, meaning only files that
aren't being touched by any PR with activity in the past 2 months.
This is to avoid conflicts for PRs that might soon be merged.
Later we can do a full treewide reformat to get the rest,
which should not cause as many conflicts.
A CI check has already been running for some time to ensure that new and
already-formatted files are formatted, so the files being reformatted here
should also stay formatted.
This commit was automatically created and can be verified using
nix-build a08b3a4d19.tar.gz \
--argstr baseRev b32a094368
result/bin/apply-formatting $NIXPKGS_PATH
In preparation for the deprecation of `stdenv.isX`.
These shorthands are not conducive to cross-compilation because they
hide the platforms.
Darwin might get cross-compilation for which the continued usage of `stdenv.isDarwin` will get in the way
One example of why this is bad and especially affects compiler packages
https://www.github.com/NixOS/nixpkgs/pull/343059
There are too many files to go through manually but a treewide should
get users thinking when they see a `hostPlatform.isX` in a place where it
doesn't make sense.
```
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv.is" "stdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv'.is" "stdenv'.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "clangStdenv.is" "clangStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "gccStdenv.is" "gccStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenvNoCC.is" "stdenvNoCC.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "inherit (stdenv) is" "inherit (stdenv.hostPlatform) is"
fd --type f "\.nix" | xargs sd --fixed-strings "buildStdenv.is" "buildStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "effectiveStdenv.is" "effectiveStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "originalStdenv.is" "originalStdenv.hostPlatform.is"
```
Added a cross compilation test for wasm32-unknown-unknown. This also
required using crate.metadata instead of using the regex to get rid of
the hash in the library filename. It also required adding a mkCrate
argument to assertOutputs so we can override the buildRustCrate used.
In order to allow for the new `cargo::` prefix for build script outputs
we have to adjust the configure-crate bash scripts in buildRustCrate to
properly parse the new additional syntax.
These changes don't affect existing build scripts configured with the
old `cargo:` prefix.
For more information, see https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script
Switched to a common attribute for library file extensions:
de70971c90
This makes buildRustCrate evaluate successfully when using
pkgsCross/pkgsStatic.
As @lopsided98 points out in #105305, since the hashes are now target
sensative, and until we find reason to actually care to test what they
are exactly, we are best just normalizing them away in the tests.
Cargo sets `CARGO_FEATURE_*` for all features when running a build
script:
https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts
Some crates have build scripts (e.g. openblas-src) that rely on the
feature variables being properly set.
Since we now need several representations of features, this change
also updates `createFeatures` to be a list of features, rather than
`rustc` feature arguments. `configureCrate` and `buildCrate` then
build the required representations as-needed.
Fixes#68978
As it turns out Darwin does most of the things differently then "normal"
systems. They are using a different shared library extension and require
an obscure commandline parameter that has to be added to every build
system out there. That issue seems to be with clang on Darwin as on
Linux that flag isn't required to build the very same tests (when using
clang).
After adjusting these two details the tests are running fine on the
darwin box that I was able to obtain.
