This commit was created by a combination of scripts and tools:
- an ast-grep script to prefix things in meta with `lib.`,
- a modified nixf-diagnose / nixf combination to remove unused `with lib;`, and
- regular nixfmt.
Co-authored-by: Wolfgang Walther <walther@technowledgy.de>
As documented in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#abi-tag.
This is useful to check whether a wheel is compatible with a certain interpreter. [Pyproject.nix](https://github.com/pyproject-nix/pyproject.nix) has [functions to perform wheel compatibility checking](https://pyproject-nix.github.io/pyproject.nix/lib/pypa.html#function-library-lib.pypa.isWheelFileCompatible) against a Python interpreter, and has computed interpreter ABI tags itself.
The recent addition of free threading (`python313FreeThreading`) complicates this by not being introspectable:
A GIL Python (non free-threaded) has an ABI tag `cp313` while the free-threaded Python has `cp313t`, but the package doesn't communicate whether `enableGIL` is true or false, leaving no way to compute the tag.
The same goes for if debugging support was added to the derivation: A `d` suffix would need to be added.
Additionally ABI tags has no defined format and can really only be accurately computed by having insight into how the ABI tags are used by a specific interpreter, meaning that the only correct place to compute ABI tags is within the context of a particular Python derivation.
While this has no immediate use within nixpkgs it could be used as a basis to provide compatibility assertions regarding wheel compat at eval time.
Without the change the eval fails as:
$ nix build --no-link -f. pythonInterpreters.pypy310_prebuilt.tests.test-packageOverrides
error:
… in the condition of the assert statement
at pkgs/development/interpreters/python/tests.nix:212:11:
211| in
212| assert myPython.pkgs.foobar == myPython.pkgs.numpy;
| ^
213| myPython.withPackages (ps: with ps; [ foobar ]);
… while evaluating the attribute 'outPath'
at lib/customisation.nix:421:7:
420| drv.drvPath;
421| outPath =
| ^
422| assert condition;
(stack trace truncated; use '--show-trace' to show the full, detailed trace)
error: attribute 'mainProgram' missing
at pkgs/by-name/me/meson/package.nix:75:52:
74| --replace-fail "PythonExternalProgram('python3', mesonlib.python_command)" \
75| "PythonExternalProgram('${python3.meta.mainProgram}', mesonlib.python_command)"
| ^
76| substituteInPlace mesonbuild/modules/python3.py \
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](8616af08d9/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
In preparation for the deprecation of `stdenv.isX`.
These shorthands are not conducive to cross-compilation because they
hide the platforms.
Darwin might get cross-compilation for which the continued usage of `stdenv.isDarwin` will get in the way
One example of why this is bad and especially affects compiler packages
https://www.github.com/NixOS/nixpkgs/pull/343059
There are too many files to go through manually but a treewide should
get users thinking when they see a `hostPlatform.isX` in a place where it
doesn't make sense.
```
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv.is" "stdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv'.is" "stdenv'.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "clangStdenv.is" "clangStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "gccStdenv.is" "gccStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenvNoCC.is" "stdenvNoCC.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "inherit (stdenv) is" "inherit (stdenv.hostPlatform) is"
fd --type f "\.nix" | xargs sd --fixed-strings "buildStdenv.is" "buildStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "effectiveStdenv.is" "effectiveStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "originalStdenv.is" "originalStdenv.hostPlatform.is"
```
The way we build python environments is subtly broken. A python
environment should be semantically identical to a vanilla Python
installation in, say, /usr/local. The current implementation, however,
differs in two important ways. The first is that it's impossible to use
python packages from the environment in python virtual environments. The
second is that the nix-generated environment appears to be a venv, but
it's not.
This commit changes the way python environments are built:
* When generating wrappers for python executables, we inherit argv[0]
from the wrapper. This causes python to initialize its configuration
in the environment with all the correct paths.
* We remove the sitecustomize.py file from the base python package.
This file was used tweak the python configuration after it was
incorrectly initialized. That's no longer necessary.
The end result is that python environments no longer appear to be venvs,
and behave more like a vanilla python installation. In addition it's
possible to create a venv using an environment and use packages from
both the environment and the venv.
The nixpkgs-unstable channel's programs.sqlite was used to identify
packages producing exactly one binary, and these automatically added
to their package definitions wherever possible.
When PyPy introduced Python 3.8 support with version 7.3.6, they also
migrated to using CPython's directory layout [0]:
> The 3.8 package now uses the same layout as CPython, and many of the
PyPy-specific changes to `sysconfig`, `distutils.sysconfig`, and
`distutils.commands.install.py` have been removed. The stdlib now is
located in `<base>/lib/pypy3.8` on `posix` systems...
When we upgraded past this version and added Python 3.8 support [1], the
`sitePackages` value was never updated, leading `bootstrapped-pip` to fail
to build, because wheel was trying to be located in `$out/site-packages`,
when it was actually installed to `$out/lib/pypy3.8/site-packages`.
[0]: https://www.pypy.org/posts/2021/10/pypy-v736-release.html
[1]: eec28b8cfd
- Remove code related to i686 archs (not supported anymore since #209100)
- Split non-install steps in installPhase in installCheckPhase/preFixup
- Add missing pre/post hooks
Also build it with pythonInterpreters.pypy27_prebuilt instead of
python27, to avoid it being broken after #201859 is merged.
Keep in mind that the above change will make this derivation only work
with x86_64-linux for now (since pypy27_prebuilt is only packaged for
that platform). However support for newer platforms can be added later,
and a working package for one platform is better than a broken package
in 4 platforms.
