When pkgsStatic is already musl, we can let pkgsMusl simply be a no-op.
As the interaction of pkgsStatic and pkgsMusl machinery is currently
broken, this fixes pkgsStatic.pkgsMusl.stdenv.
`appendOverlays []` returned `self` (the unspliced inner fixpoint)
while `import nixpkgs {}` returns `pkgs.__splicedPackages` (the
spliced version). This caused `pkgs.makeWrapper` and similar
spliced packages to lose their `__spliced` attribute after passing
through `appendOverlays`, breaking cross-compilation in NixOS
modules.
The NixOS `nixpkgs.nix` module calls `cfg.pkgs.appendOverlays
cfg.overlays`, and when no overlays are defined, the empty-list
short-circuit returned unspliced packages. This meant
`nativeBuildInputs = [ pkgs.makeWrapper ]` in NixOS modules would
evaluate the host-platform makeWrapper (which throws in cross
context) instead of being spliced to buildPackages.makeWrapper.
Fix by returning `self.__splicedPackages` instead of `self` for
the empty-list case.
this was done manually, fixing the eval and any remaining stuff of the previous commit
together with the previous commit this shouldn't create any rebuilds
For the regular user, this still throws early whenever an attribute
tries to use `pkgsi686Linux`. For CI, this doesn't throw, but instead
overwrites `mkDerivation` to return only derivations marked as
`meta.broken`. This propagates to all reverse dependencies and allows
filtering out these attributes nicely on unsupported platforms.
this equates to -fno-strict-aliasing
this is not a particularly new flag and is supported by all our
gcc and clang versions.
add to pkgsExtraHardening's defaultHardeningFlags.
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](8616af08d9/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
Darwin includes its SDK in `extraBuildInputs`, which isn’t needed in
`stdenvNoCC` because it depends on a compiler. Including it greatly
increases the size of the `stdenvNoCC` closure (by ~1 GiB).
This wasn’t an issue in the past when darwin included CoreFoundation
because CoreFoundation on its own is very small. Now that it includes
the whole SDK and links some bintools, it pulls in 300~400 MiB for the
SDK plus another ~445 MiB for LLVM.
Fixes#371465
When using pkgsCross with a system that ends up the same as the
localSystem, then modifications for package sets like pksgMusl need to
be done for **both** localSystem and crossSystem. Consider the following
on x86_64-linux:
pkgsCross.gnu64.pkgsMusl
Before this change, this would result in a musl buildPlatform, but a gnu
hostPlatform. This breaks the promise of "stacking" package sets on top
of each other.
After this change, it results in a musl buildPlatform and a musl
hostPlatform. This works better.
One could expect this to result in the same as pkgsCross.musl64, i.e. a
gnu buildPlatform and a musl hostPlatform, however I couldn't get this
to work without increasing memory usage for ci/eval by many, many GB.
This is caused by usage of pkgsi686Linux inside the main package set,
which follows the same hybrid pattern.
The various pkgsXYZ top-level package sets did not pass localSystem /
crossSystem to lower levels, so far. This change propagates original
arguments to lower levels, which include the overrides made by an upper
package sets.
There is an extensive test-suite to test various combinations of package
sets in pkgs/test/top-level. There are a few basic promises made:
- Package sets must be idempotent. pkgsMusl.pkgsMusl === pkgsMusl.
- Once pkgsCross is used any subsequent package sets should affect the
**host platform** and not the build platform. Examples:
- pkgsMusl.pkgsCross.aarch64-multiplatform is a cross compilation from
musl to glibc/aarch64
- pkgsCross.aarch64-multiplatform.pkgsMusl is a cross compilation to
musl/aarch64
- Modifications from an earlier layer should not be lost, unless
explicitly overwritten. Examples:
- pkgsStatic.pkgsMusl should still be static.
- pkgsStatic.pkgsCross.gnu64 should be static, but with glibc instead
of musl.
Exceptions / TODOs:
- pkgsExtraHardening is currently not idempotent, because it applies the
same flags over and over again.
Supersedes #136549Resolves#114510Resolves#212494Resolves#281596
No change, just move appendOverlays and extend to the bottom, since they
will be changed much less often. This makes it easier to compare the
other package sets side-by-side.
pkgsLLVM introduced an unwanted target offset, as can be seen with the
following commands.
Before this commit:
```
$ nix eval --system x86_64-linux -f . pkgsCross.aarch64-multiplatform.buildPackages.stdenv.hostPlatform.config
"x86_64-unknown-linux-gnu"
$ nix eval --system x86_64-linux -f . pkgsCross.aarch64-multiplatform.buildPackages.pkgsLLVM.stdenv.hostPlatform.config
"aarch64-unknown-linux-gnu"
Explaination: These two should be the same, since the only difference
between them is the addition of "pkgsLLVM". The buildPackages of the
pkgsCross set should have a hostPlatform which is the native platform.
```
After this commit:
```
$ nix eval --system x86_64-linux -f . pkgsCross.aarch64-multiplatform.buildPackages.stdenv.hostPlatform.config
"x86_64-unknown-linux-gnu"
$ nix eval --system x86_64-linux -f . pkgsCross.aarch64-multiplatform.buildPackages.pkgsLLVM.stdenv.hostPlatform.config
"x86_64-unknown-linux-gnu"
```
When elaborating a system with both "config" and "system" arguments
given, they might not match the parsed results. Example:
elaborate {
config = "i686-unknown-linux-gnu";
system = "x86_64-linux";
}
This would result in a parsed system for i686, because the config
argument is preferred. But since "// args //" comes after system has
been inferred from parsed, it is overwritten again. This results in
config and parsed all pointing to i686, while system still tells the
story of x86_64.
Inconsistent arguments can also be given when passing "parsed" directly.
This happened in stage.nix for the various package sets.
The solution is simple: One of the three arguments needs to be treated
as the ultimate source of truth. "system" can already be losslessly
extracted from "parsed". However, "config" currently can not, for
example for various -mingw32 cases. Thus everything must be derived
from "config".
To do so, "system" and "parsed" arguments are made non-overrideable for
systems.elaborate. This means, that "system" will be used to parse when
"config" is not given - and "parsed" will be ignored entirely.
The systemToAttrs helper is exposed on lib.systems, because it's useful
to deal with top-level localSystem / crossSystem arguments elsewhere.
In preparation for the deprecation of `stdenv.isX`.
These shorthands are not conducive to cross-compilation because they
hide the platforms.
Darwin might get cross-compilation for which the continued usage of `stdenv.isDarwin` will get in the way
One example of why this is bad and especially affects compiler packages
https://www.github.com/NixOS/nixpkgs/pull/343059
There are too many files to go through manually but a treewide should
get users thinking when they see a `hostPlatform.isX` in a place where it
doesn't make sense.
```
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv.is" "stdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv'.is" "stdenv'.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "clangStdenv.is" "clangStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "gccStdenv.is" "gccStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenvNoCC.is" "stdenvNoCC.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "inherit (stdenv) is" "inherit (stdenv.hostPlatform) is"
fd --type f "\.nix" | xargs sd --fixed-strings "buildStdenv.is" "buildStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "effectiveStdenv.is" "effectiveStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "originalStdenv.is" "originalStdenv.hostPlatform.is"
```
For a long time, we've had `crossLibcStdenv`, `*Cross` libc attributes,
and `*bsdCross` pre-libc package sets. This was always bad because
having "cross" things is "not declarative": the naming doesn't reflect
what packages *need* but rather how we *provide* something. This is
ugly, and creates needless friction between cross and native building.
Now, almost all of these `*Cross` attributes are gone: just these are
kept:
- Glibc's and Musl's are kept, because those packages are widely used
and I didn't want to risk changing the native builds of those at this
time.
- generic `libcCross`, `theadsCross`, and friends, because these relate
to the convolulted GCC bootstrap which still needs to be redone.
The BSD and obscure Linux or freestnanding libcs have conversely all
been made to use a new `stdenvNoLibc`, which is like the old
`crossLibcStdenv` except:
1. It usable for native and cross alike
2. It named according to what it *is* ("a standard environment without
libc but with a C compiler"), rather than some non-compositional
jargon ("the stdenv used for building libc when cross compiling",
yuck).
I should have done this change long ago, but I was stymied because of
"infinite recursions". The problem was that in too many cases we are
overriding `stdenv` to *remove* things we don't need, and this risks
cyles since those more minimal stdenvs are used to build things in the
more maximal stdenvs.
The solution is to pass `stage.nix` `stdenvNoCC`, so we can override to
*build up* rather than *tear down*. For now, the full `stdenv` is also
passed, so I don't need to change the native bootstraps, but I can see
this changing as we make things more uniform and clean those up.
(adapted from commit 51f1ecaa59)
(adapted from commit 1743662e55)
