This is in public preview now. These runners run in a docker container
with only a single vCPU instead of 4 like the other jobs. For most of
our jobs, this should be plenty, except for eval and linting.
If cachix fails to install, we don't actually need to error out - we can
still continue at the cost of building stuff more than required. That's
better than erroring out, just because cachix has another hiccup,
especially for the Merge Queue.
Since it was previously possible, in theory, to extract the
CACHIX_AUTH_TOKEN via Nix eval on untrusted inputs, this rotates the
token and clears the cache - and while on it moves to a new cache, that
is owned by a newly created nixpkgs-ci org instead of an individual.
Evaluating untrusted code in the presence of secrets is unsafe in
general, thus we only provide the cachix auth token when these jobs run
in the merge queue. This is enough for all practical purposes, PRs will
be able to pull stuff from cachix that was built in the Merge Queue
previously.
Instead of setting the maintainer-related labels based entirely on Eval
results, this uses the new maintainer map from the target branch. This
allows labeling PRs correctly, that had been created *before* a
contributor became a maintainer of the respective package.
This creates a simple mapping of all packages to github ids of all their
maintainers. This is uploaded as an artifact and is then available for
download on the latest commit of each branch with a merge queue. This
will allow scheduled jobs to use this information for setting
maintainer-related labels, to request reviewers and to implement the
merge-bot.
The advantage over querying this information directly via Nix in each
case: The scheduled job does not need to install Nix and does not need
to checkout the target branch.
Compared to obtaining the maintainer information just for a single PR
during Eval, this will allow retroactively changing maintainers for a
package: For example, it allows to request a new maintainer as reviewer
for a PR that was created before they became maintainer, but is still
open - and similarly for maintainer labels and merge-bot rights.
None of these extensions are implemented by this PR, yet.
This runs the compare step in the merge queue, to get a combined diff
against the previous queue item. This can be used as a base for multiple
different features in the future, for example preventing accidental mass
rebuilds in the merge queue, or diffing of packages' meta.
Once we start running similar jobs in the PR and Merge Queue workflows,
the Test workflow will run into a problem: Both nested workflows will
try to upload the same artifacts, which errors out.
By providing a prefix to use for artifact names, this will be a
non-issue when we later run more of Eval in the merge queue or when we
add the build jobs to the queue as well.
The Eval report which tests performance between Nix/Lix versions on
update of `ci/pinned` wrongly returned errors, when only the special
attribute `release-checks` changed. Since this reads in all of Nixpkgs,
it will change with any formatting change that is introduced at the same
time via update of any of `treefmt`'s formatters.
This change runs Lix on the target commit and Nix on the merged commit.
This does two things for us at once:
- We test both Lix and Nix.
- We ensure that both Lix and Nix produce the same output hashes.
If Lix and Nix were to produce different output hashes at some point,
this would show up as rebuilds in every PR.
This was only separate to work around possible delays from the target
branch's eval workflow. With the switch to the merge queue, this delay
is impossible - the relevant target commit will only appear once Eval
has completed in the merge queue, so Eval will be guaranteed to have
finished.
By running attrpaths and outpaths in the same step, we share ~10 seconds
of eval time, traversing through all of Nixpkgs.
Running eval in the merge queue prevents eval failures after merging
multiple PRs with semantic merge conflicts.
It's also the base for allowing more sophisticated checks about rebuild
counts in the merge queue later.
When branches are directly pushed to, such as for periodic merges, this
will not run Eval immediately. However, the next PR will run it as part
of its Eval step and will cache the results. Any further PR will then
benefit from the same caching again.
This also avoids running Eval twice when pushing to staging-next or
haskell-updates, where a PR is open at the same time. Here, the PR's
Eval still runs on the push, of course.
Due to how we pass in existing store paths via CLI arguments for the
diff and combine scripts, Nix didn't register a dependency on the store
paths properly. This meant that some of the derivations that were built,
didn't have the right store paths made available in the sandbox -
leading to all kinds of "not found" errors.
We worked around this in CI by resolving the symlinks to the nix store
beforehand. We tried to work around this locally by storing the nix
store path in BASELINE, but this didn't fully work. By explicitly
registering these store paths as dependencies, this should work across
the board - without any magic required by the caller.
Both `nix flake check` and `nix-env` are single-threaded, so no reason
to serialize their calls and waste time.
This brings down the runtime for this job from ~1:51 to ~1:27 in my
tests.
Instead of uploading the outpaths as artifact, this uploads them via
cachix. Most of all, this makes CI less brittle, because Eval in PRs
will still be able to succeed, even if no workflow run for the push
event could be found on the target branch. It will just take longer.
This also makes moving Eval into the Merge Queue easier to do: When
downloading artifacts from a different run, these would always have to
match on the right event, too. By pulling from cachix, the same workflow
can support target branches with merge queue and without merge queue at
the same time. The latter would still use the push event, while the
former could use the merge_group event.
Last but not least, this should fix Eval on PRs targeting `wip-`
branches and any other branches that the push event doesn't trigger on.
These would never find an Eval result from the target branch and could
never show rebuilds accurately. Now these PRs should work at a slightly
higher runtime cost.
With the latest performance improvements, this only adds up to 1s, but
reduces complexity in managing pinned checkouts explicitly, especially
when we need *both* pinned checkouts in the next commit.
Instead of fetching up to 3 times on each new checkout, we now fetch all
the commits we're going to need at once. Afterwards, we checkout the
different worktrees in parallel, which doesn't give us much, yet,
because it would still be IO-bound on its own. Inconsistent IO
performance on disk is also the biggest limitation for checkout right
now, where checkout times range everywhere from 20s to 40s.
By checking out the worktrees on a tmpfs, the actual checkout only takes
1s and benefits from parallelization. The overall checkout time is now
8-11s, depending on the number of commits.
That's a reduction of 10-30s and we get this speedup for almost every
job in the PR workflow, which is huge.
This potentially has a nice side-effect for Eval, too: Because the repo
is in RAM, Eval seems to run slightly faster, up to 10 seconds less.
This moves the logic to "check whether the PR can be merged and
determine the relevant commits" into the PR / prepare job directly -
since that's the only place where it is supposed to be used.
Because of the if condition in get-merge-commit, this logic was run
multiple times, for example in the lint workflow, where only one of
targetSha or mergedSha was provided as input. However, this input was
thrown away directly. This might not be a big problem, because this was
not expensive, so far. But with the next commit, this will become more
so.
This also separates the logic a bit cleaner - `prepare` figures out all
the parameters for the whole PR workflow, while `checkout` handles the
consistency around these checkouts.
The conclusion of the `versions` job propagates from through `eval` to
`compare`, which meant the `compare` job was skipped. No rebuild labels,
no reviewer requests.
Also, we don't want to run eval when `versions` runs, but fails.
With this change, we start running Eval on all available Lix and Nix
versions. Because this requires a lot of resources, this complete test
is only run when `ci/pinned.json` is updated.
The resulting outpaths are checked for consistency with the target
branch. A difference will cause the `report` job to fail, thus blocking
the merge, ensuring Eval consistency for Nixpkgs across different
versions.
This implements a kind of "ratchet style" check: Since we originally
confirmed that the versions currently in Nixpkgs at the time of this
commit match Eval behavior of Nix 2.3, we can ensure consistency with
Nix 2.3 down the road, even without testing for it explicitly.
There had been one regression in Eval consistency for Nix between 2.18
and 2.24 - two tests in `tests.devShellTools` produce different results
between Lix 2.91+ (which was forked from Nix 2.18) and Nix 2.24+. I
assume it's unlikely that such a change would be "fixed" by now, thus I
added an exception for these.
As a bonus, we also present the total time in seconds it takes for Eval
to complete for every tested version in a summary table. This allows us
to easily see performance improvements for Eval due to version updates.
At this stage, this time only includes the "outpaths" step of Eval, but
not the generation of attrpaths beforehand.
The Dependabot update change the hashes to the latest main branch commit
instead of the v5.0.0 tag - also it didn't adjust the tags in the
comments accordingly. Last but not least, one of the references used a
`@v5` reference instead of the commit hash. The latter is probably what
Dependabot tripped on.
This is slightly faster than downloading and extracting a tarball and
additionally allows a sparse checkout. No need to download docs or nixos
for our purpose.
The data is quite noisy, but suggests improvements from anywhere between
5-15 seconds for each job using the pinned nixpkgs.
