With this argument fetchgit will make a subdirectory of the Git
repository a root of the resulting store path. This is helpful for
dealing with monorepos where many projects are in separate directories
and don't need a new source hash every time the monorepo is updated.
Commit hash is removed from the name of the derivation to prevent it
from changing the store path when nothing in the subdirectory changes.
With this argument nix-prefetch-git will make a subdirectory of the Git
repository a root of the resulting store path. This is helpful for
dealing with monorepos where many projects are in separate directories
and don't need a new source hash every time the monorepo is updated.
The corresponding fetchgit change follows.
For projects that have `submodule.$name.update` set to something other
than the default value of "checkout", this change will ensure those
submodules are fetched as well.
This patch adds `lib.repoRevToName` function that generalizes away most of the
code used for derivation name generation by `fetch*` functions (`fetchzip`,
`fetchFromGitHub`, etc, except those which are delayed until latter commits
for mass-rebuild reasons).
It's first argument controls how the resulting name will look (see below).
Since `lib` has no equivalent of Nixpkgs' `config`, this patch adds
`config.fetchedSourceNameDefault` option to Nixpkgs and then re-exposes
`lib.repoRevToName config.fetchedSourceNameDefault` expression as
`pkgs.repoRevToNameMaybe` which is then used in `fetch*` derivations.
The result is that different values of `config.fetchedSourceNameDefault` now
control how the `src` derivations produced by `fetch*` functions are to be
named, e.g.:
- `fetchedSourceNameDefault = "source"` (the default):
```
$ nix-instantiate -A fuse.src
/nix/store/<hash>-source.drv
```
- `fetchedSourceNameDefault = "versioned"`:
```
$ nix-instantiate -A fuse.src
/nix/store/<hash>-libfuse-2.9.9-source.drv
```
- `fetchedSourceNameDefault = "full"`:
```
$ nix-instantiate -A fuse.src
/nix/store/<hash>-libfuse-2.9.9-github-source.drv
```
See the documentation of `config.fetchedSourceNameDefault` for more info.
Added fetchTags feature to fetchgit, explicit and clear support for
fetching all tags after the source tree fetch completes. Doing this at
build-time in the fetcher is required for packages that invoke commands
like 'git describe' which require tags, and since the nix store is
read-only by design, it is not possible to git fetch --tags at
activation- or run-time. This feature may have been possible by
specifying a postFetch option including calling git fetch --tags,
however doing so obfuscates the solution to this very real problem.
Explicit support for fetching tags should be a first class citizen just
like fetching other refs.
This allows running code to change the environment before the
nix-prefetch-git script is run. In particular, it allows setting things
like NIX_PREFETCH_GIT_CHECKOUT_HOOK, which allows additional code to be
run before deleting the .git directory.
This also means there's potentially value in performing a deep clone
then removing the .git directory, so remove the assertion that prevents
doing so.
This allows tools like `npins` to avoid re-downloading pre-fetched sources at evaluation time,
by setting the same symbolic store path name as `builtins.fetchGit` does,
and would make hacks such as [0] obsolete.
Historical note: `builtins.fetchGit` was introduced in 2016 [1] and
released in 2018 [2]. Before that, there was only the Nixpkgs `fetchgit`,
and the convention for using constant store path names [3] was not established yet [4].
While arguably in retrospect store path names were a mistake, we cannot
undo it here even partially by setting them to a constant, as it would invalidate
countless fixed-output derivations and thus incur an increase in
resource consumption and potentially a large number of build failures.
[0]: f0449d090c
[1]: 38539b943a
[2]: https://github.com/NixOS/nix/releases/tag/2.0
[3]: db00265e81/source/guides/best-practices.md (L263-L264)
[4]: a8603605aa
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](8616af08d9/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
Git v2.49.0 added support for using the faster zlib-ng library, and
benchmarks by GitLab show it's notably faster, so use it for our builds
too.
The different zlib implementation does change the contents of the .git
directory, which means the hashes for fetchgit tests that leave the .git
directory in place are expected to change.
https://about.gitlab.com/blog/2025/03/14/whats-new-in-git-2-49-0/#introduction-of-zlib-ng
The deepClone argument to fetchgit implies leaveDotGit. Make the
"submodule-deep" fetchgit test more useful by having it remove the .git
directory, as recommended in the fetchgit documentation, so it is
distinguished from the "submodule-leave-git-deep" test.
This version change includes a change to the contents of the .git
directory, which results in changes to hashes for any fetchgit
invocations that set leaveDotGit to true. Per the fetchgit
documentation and discussions in #372784, the contents of the .git directory are not stable, so
anyone using that should either only be using it for temporary testing
purposes, or should be removing the .git directory in fetchgit's
postFetch phase, so before the hash is calculated. That means there's
no need to do any other handling here to avoid compatibility problems,
and can just update the relevant hashes in the fetchgit tests.
There's still a chance this will break some other builds because of the
change to the contents of the .git directory. If so, whoever's
responsible for those other derivations should either stop using
leaveDotGit, or should follow the documented advice to remove the .git
directory in the postFetch phase.
It's hard to put the sourcing of ./.attrs.sh into all builder
consistently - mistakes will happen. Thus, load structured attrs once in
make-derivation and then source the remaining builder on top.
This should fix quite a few builders with structured attributes in
principle. Most importantly it helps substitute / substituteAll, which
are required for bootstrap on some platforms.
Since nix 2.20, `nix-store --add-fixed` doesn't accept paths where the
parent directory is a symlink. On macOS, /tmp is a symlink to
/private/tmp, which causes a "'/tmp' is a symlink" error:
```
$ nix run github:nixos/nixpkgs/24.11-beta#nix-prefetch-git -- --url https://github.com/IFTTT/polo.git --rev 316aa2ac210a45a7fc400ab921831493d5dd21b8 --hash sha256
Initialized empty Git repository in /private/tmp/git-checkout-tmp-1Bf9bIv7/polo-316aa2a/.git/
remote: Enumerating objects: 51, done.
remote: Counting objects: 100% (51/51), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 51 (delta 8), reused 19 (delta 5), pack-reused 0 (from 0)
Unpacking objects: 100% (51/51), 19.57 KiB | 541.00 KiB/s, done.
From https://github.com/IFTTT/polo
* branch HEAD -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...
error: path '/tmp' is a symlink
```
Avoid this by resolving /tmp to a real directory in all the prefetch scripts
It's become a common pattern to use `rev = "refs/tags/${version}"` rather than
just `rev = version` to ensure that the tag gets fetched rather than a branch
that has the same name. This has so far been done using boilerplate though, so
let's add a simple abstraction to fetch a tag instead.
This envvar is also added to lib.proxyImpureEnvVars since it's
typically required for https proxies.
This change also updates fetchgit and go module fetching to use this
envvar. NIX_GIT_SSL_CAINFO is still supported for backwards
compatibility in fetchgit.
Some programs (e.g. nvfetcher) parse the output of nix-prefetch-git as
JSON. However, there is a pushd/popd command in nix-prefetch-git that
outputs the directory name, breaking the structure of the result JSON.
We suppressed it with `>/dev/null`.
Passingly fixes a mixuse of tab and spaces.
This script needs to support being run both as part of a `fetchgit`
derivation and as a standalone, command-line tool.
The use of `$NIX_BUILD_TOP` only works when used in `fetchgit` but not when
invoked as a standalone tool. Instead we try to respect `$NETRC` so that
the command-line invocation behaves more like standard tools and the
`fetchgit` derivation can explicitly set `$NETRC` when `netrcPhase` is used
to avoid all ambiguity.
This fixes a regression introduced in https://github.com/NixOS/nixpkgs/pull/256628
which broke fetching with private = true through a netrc file.
Tested locally with a really special github enterprise.
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into
builtins.derivation {
builder = stdenv.shell;
args = [ "-e" builder ];
}
This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].
Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:
* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
(see previous commit for more context), so prefer the environment
variable if possible.
* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
can happen.
* If neither applies, it can be assumed that `__structuredAttrs` is
turned off and thus nothing needs to be done.
[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
it can point to a wrong location on older Nix versions with a bug in
`__structuredAttrs`.
nix-prefetch-git is either run as part of a build, usually sandboxed,
or outside a build, unsandboxed, to prefetch something that will later
be used in a build. It's important that the latter use produces
hashes that can be reproduced by the former.
One way that they can differ is if the user's git config does
something that changes the result of git clone. I ran into this,
because my global git config automatically enables git-lfs, whereas
nix-prefetch-git otherwise only uses git-lfs if specifically
requested. This led to very confusing hash mismatches.
