npm's locking process resolves Git dependencies to a given commit, separated
by a `#`. Before this change, any encounter of these URLs would cause `fetchGit`
to fail, as they're not valid repo URLs.
Up to now, existing symlinks were replaced by removing the existing link
and then creating a new one. This left a window where the link did not
exist, which would break cases where the hook was running concurrently
with other things using node_modules.
In my case, I would run multiple webpack builds at once like:
nix develop -c webpack build --mode production
nix develop -c webpack build --mode development
If one of the shells was ready faster than the other, webpack would
start running, then the second would run the linkNodeModules hook, and
the earlier webpack build would run into missing files as the second
linkNodeModules hook removed symlinks to replace them.
This change mitigates this race condition in two ways:
- Not replacing the symlink if it already points to the right target
(this also avoids superfluous filesystem writes)
- Replacing the symlink atomically, using a rename,
otherwise. The symlink was replaced atomically prior to
db7050bb88 as well, but the name of the
temporary symlink was always the same, which led to a similar race
condition. This change reintroduces the atomic replacement, but adds
the PID to the temporary filename in order to avoid conflicting with
other instances of the hook running concurrently.
It's currently possible to run into a race condition when entering the same development environment concurrently:
```
❯ git fetch; jj rebase -b 'all:mutable() & mine()' -d main@origin --skip-emptied
Rebased 4 commits onto destination
Abandoned 1 newly emptied commits
Working copy (@) now at: tnyknvqt 93e36def (empty) (no description set)
Parent commit (@-) : wnqxqyyl e0983a05 main@origin | Increase limit of max number of words
Added 0 files, modified 8 files, removed 0 files
direnv: loading ~/dev/REDACTED/.envrc
direnv: loading https://raw.githubusercontent.com/nix-community/nix-direnv/3.0.4/direnvrc (sha256-DzlYZ33mWF/Gs8DDeyjr8mnVmQGx7ASYqA5WlxwvBG4=)
Executing linkNodeModulesHook
node:internal/fs/promises:782
return await PromisePrototypeThen(
^
Error: ENOENT: no such file or directory, rename 'node_modules/.bin-nix-hook-temp' -> 'node_modules/.bin'
at async Object.rename (node:internal/fs/promises:782:10)
at async /nix/store/ps9ivjjxzi0fks67j6vd4gbw5dcnhp0w-link-node-modules.js:84:7
at async Promise.all (index 0)
at async main (/nix/store/ps9ivjjxzi0fks67j6vd4gbw5dcnhp0w-link-node-modules.js:58:3) {
errno: -2,
code: 'ENOENT',
syscall: 'rename',
path: 'node_modules/.bin-nix-hook-temp',
dest: 'node_modules/.bin'
}
Node.js v20.17.0
Finished executing linkNodeModulesShellHook
```
This change removes the intermediate file creation and simply tries to create a symlink directly.
If the target `node_modules/foo` already exists we unlink it and try to create the symlink again.
node_modules/.bin/ is expected to be (symlink to) a directory rather
than a regular file, so we check for that.
This seems to work for pre-existing directories as well as those
we link to the store (managed & unmanaged in the hooks terminology).
I am not entirely sure whether the original check was simply a typo
or intentional for some use-cases.
After final improvements to the official formatter implementation,
this commit now performs the first treewide reformat of Nix files using it.
This is part of the implementation of RFC 166.
Only "inactive" files are reformatted, meaning only files that
aren't being touched by any PR with activity in the past 2 months.
This is to avoid conflicts for PRs that might soon be merged.
Later we can do a full treewide reformat to get the rest,
which should not cause as many conflicts.
A CI check has already been running for some time to ensure that new and
already-formatted files are formatted, so the files being reformatted here
should also stay formatted.
This commit was automatically created and can be verified using
nix-build a08b3a4d19.tar.gz \
--argstr baseRev b32a094368
result/bin/apply-formatting $NIXPKGS_PATH
`importNpmLock.buildNodeModules` returns a derivation with a pre-built `node_modules` directory, as imported by `importNpmLock`.
This is to be used together with `importNpmLock.hooks.linkNodeModulesHook` to facilitate `nix-shell`/`nix develop` based development workflows:
```nix
pkgs.mkShell {
packages = [
importNpmLock.hooks.linkNodeModulesHook
nodejs
];
npmDeps = importNpmLock.buildNodeModules {
npmRoot = ./.;
inherit nodejs;
};
}
```
will create a development shell where a `node_modules` directory is created & packages symlinked to the Nix store when activated.
This code is adapted from https://github.com/adisbladis/buildNodeModules
This is an alternative to `fetchNpmDeps` that is notably different in that it uses metadata from `package.json` & `package-lock.json` instead of specifying a fixed-output hash.
Notable features:
- IFD free.
- Only fetches a node dependency once. No massive FODs.
- Support for URL, Git and path dependencies.
- Uses most of the existing `npmHooks`
`importNpmLock` can be used _only_ in the cases where we need to check in a `package-lock.json` in the tree.
Currently this means that we have 13 packages that would be candidates to use this function, though I expect most usage to be in private repositories.
This is upstreaming the builder portion of https://github.com/adisbladis/buildNodeModules into nixpkgs (different naming but the code is the same).
I will archive this repository and consider nixpkgs the new upstream once it's been merged.
For more explanations and rationale see https://discourse.nixos.org/t/buildnodemodules-the-dumbest-node-to-nix-packaging-tool-yet/35733
Example usage:
``` nix
stdenv.mkDerivation {
pname = "my-nodejs-app";
version = "0.1.0";
src = ./.;
nativeBuildInputs = [
importNpmLock.hooks.npmConfigHook
nodejs
nodejs.passthru.python # for node-gyp
npmHooks.npmBuildHook
npmHooks.npmInstallHook
];
npmDeps = buildNodeModules.fetchNodeModules {
npmRoot = ./.;
};
}
```
