With the latest performance improvements, this only adds up to 1s, but
reduces complexity in managing pinned checkouts explicitly, especially
when we need *both* pinned checkouts in the next commit.
Instead of fetching up to 3 times on each new checkout, we now fetch all
the commits we're going to need at once. Afterwards, we checkout the
different worktrees in parallel, which doesn't give us much, yet,
because it would still be IO-bound on its own. Inconsistent IO
performance on disk is also the biggest limitation for checkout right
now, where checkout times range everywhere from 20s to 40s.
By checking out the worktrees on a tmpfs, the actual checkout only takes
1s and benefits from parallelization. The overall checkout time is now
8-11s, depending on the number of commits.
That's a reduction of 10-30s and we get this speedup for almost every
job in the PR workflow, which is huge.
This potentially has a nice side-effect for Eval, too: Because the repo
is in RAM, Eval seems to run slightly faster, up to 10 seconds less.
For PRs targeting a fast moving branch like master, it might not
actually help because the merge commit will still be different on each
run. But on slower moving branches or when testing in a fork, it can
speed things up.
Since each of these build results is only a few hundred bytes without
any dependencies, it has essentially no cost.
Not adding this to `treefmt`, because this looks at *all* the files in
the repo, so needs to re-run even when iterating on CI only.
Pushing the source path to cachix is pointless, because the only source
we're using is Nixpkgs - and that will always be available already via
checkout. No need to ever substitute it, so no need to push it either.
This allows setting up a custom cachix cache in a contributor's fork.
That's most valuable when working on CI itself, testing and interating
in a fork. It's required when working on cachix related features
directly.
This moves the logic to "check whether the PR can be merged and
determine the relevant commits" into the PR / prepare job directly -
since that's the only place where it is supposed to be used.
Because of the if condition in get-merge-commit, this logic was run
multiple times, for example in the lint workflow, where only one of
targetSha or mergedSha was provided as input. However, this input was
thrown away directly. This might not be a big problem, because this was
not expensive, so far. But with the next commit, this will become more
so.
This also separates the logic a bit cleaner - `prepare` figures out all
the parameters for the whole PR workflow, while `checkout` handles the
consistency around these checkouts.
These two jobs mostly used the same code anyway, so they can easily run
in the same job. This uses fewer resources and avoids running (and
failing) the codeowners review job on Pull Requests from development
branches like staging-next or haskell-updates.
It also opens the door for more improvements later on, for example it
will be much easier to implement a shared reviewer limit for both owners
and maintainers now.
This runs the "check owners" job as part of the main PR workflow, with
multiple advantages:
- The job doesn't run anymore when undrafting a PR, where it's not
required, because the OWNERS file's contents didn't change.
- A valid OWNERS file is now a requirement to merge a PR.
- The OWNERS file is always checked on the exact same test merge commit
that the remainder of the workflows are running on as well.
This just moves the code over to ci/github-script to make it easy to
test and iterate on locally.
The name `prepare` is chosen, because the script will be extended with
the other steps from "PR / prepare" next.
The filtered checkout we used before was a nice idea, but only worked
for small numbers of commits in a PR. It would fetch the whole history
initially, but only fetch objects on demand. This turns out to be much
too slow in a PR with 18 commits, regularly hitting the 10 minute
timeout, even when running it locally.
The new approach uses regular checkouts again. In contrast to the old
style, before we switched to the filtered checkout, this only fetches
exactly the commits we need - and all of them at once. This is
significantly faster than both other approaches, and scales much better.
A bigger number of commits doesn't have much of an effect, if any at
all.
The `cancelled()` condition seems to only apply when *the whole
workflow* was cancelled. This is not the case when a single job is
cancelled due to timeout.
We can replicate this by checking each needs.result manually.
This currently times out after 3 minutes. Give it a bit more time. 10
minutes might be excessive, but we only really want to guard against a
stuck job taking 6 hours.
This currently happens, for still unknown reasons, for the "check cherry
picks" job. The job gets cancelled by GHA mid-way. This should be the
same as an error, because an important check didn't run: Merging should
be blocked and auto-merge should not succeed.
The conclusion of the `versions` job propagates from through `eval` to
`compare`, which meant the `compare` job was skipped. No rebuild labels,
no reviewer requests.
Also, we don't want to run eval when `versions` runs, but fails.
The additional `workflows` permissions are required to backport
Dependabot updates. The permissions had been added to the app a while
ago, but we forgot to actually use them.
With this change, we start running Eval on all available Lix and Nix
versions. Because this requires a lot of resources, this complete test
is only run when `ci/pinned.json` is updated.
The resulting outpaths are checked for consistency with the target
branch. A difference will cause the `report` job to fail, thus blocking
the merge, ensuring Eval consistency for Nixpkgs across different
versions.
This implements a kind of "ratchet style" check: Since we originally
confirmed that the versions currently in Nixpkgs at the time of this
commit match Eval behavior of Nix 2.3, we can ensure consistency with
Nix 2.3 down the road, even without testing for it explicitly.
There had been one regression in Eval consistency for Nix between 2.18
and 2.24 - two tests in `tests.devShellTools` produce different results
between Lix 2.91+ (which was forked from Nix 2.18) and Nix 2.24+. I
assume it's unlikely that such a change would be "fixed" by now, thus I
added an exception for these.
As a bonus, we also present the total time in seconds it takes for Eval
to complete for every tested version in a summary table. This allows us
to easily see performance improvements for Eval due to version updates.
At this stage, this time only includes the "outpaths" step of Eval, but
not the generation of attrpaths beforehand.
The Dependabot update change the hashes to the latest main branch commit
instead of the v5.0.0 tag - also it didn't adjust the tags in the
comments accordingly. Last but not least, one of the references used a
`@v5` reference instead of the commit hash. The latter is probably what
Dependabot tripped on.
This is slightly faster than downloading and extracting a tarball and
additionally allows a sparse checkout. No need to download docs or nixos
for our purpose.
The data is quite noisy, but suggests improvements from anywhere between
5-15 seconds for each job using the pinned nixpkgs.
None of our jobs is expected to run for 6 hours, the GitHub limit. These
limits are generous and take into accounts that some jobs need to wait
for others.
If jobs exceed these times, most likely something else is wrong and
needs investigation.
This reverts commit f2648b263b.
While the idea to never use swap was fine, in practice this meant that
when nix ran OOM, some other process was killed instead. This lead to
the job not being possible to be cancelled anymore and thus needing to
timeout, before subsequent jobs could be scheduled. This can take up to
6 hours for GitHub Actions by default.
Re-enabling the swap file to catch this case more gracefully. It's still
the goal to never actually *use* the swap file during Eval and just a
safeguard.
Keeping the changed chunkSize and not reverting it - this makes it
slightly less likely to hit the swap file when running with Lix.
Recent performance tests show that (a) swapping heavily slows down the
Eval job, while (b) lowering the chunkSize does not have an effect on
run-time. It does on memory usage, though - thus we can get rid of
swapping entirely by reducing chunkSize respectively.
Introduces a basic merge queue workflow to initially only run lints.
This will avoid accidentally merging changes which break nixfmt after
its recent update to 1.0.0.
This adds a build job for the tarball, which might help uncover eval
issues on attributes not normally touched by Eval, aka those added in
`pkgs/top-level/packages-config.nix`.
