Commit Graph

179 Commits

Author SHA1 Message Date
Wolfgang Walther
f33892fcc9 ci/eval/compare/maintainers: ping by-name reviewers for pure refactors
This allows requesting reviewers for pure refactor PRs, which don't
cause a rebuild of the package. This is only possible for by-name,
because only here the package names can be inferred from the filenames.
2025-10-17 16:37:14 +02:00
Wolfgang Walther
0a5ffcdf31 ci/eval/compare/maintainers: ping by-name reviewers for all rebuilds
This adds support to ping maintainers when arbitrary files in by-name
are changed, as long as they still cause a rebuild. For example, this is
the case when changing .json files with version metadata. These were
previously not detected as belonging to the package, and didn't cause
maintainer pings.
2025-10-17 16:37:09 +02:00
Wolfgang Walther
7c865ddd46 ci/eval/compare/maintainers: refactor path matching
The only reason for the additional `lib.hasSuffix` check was, that the
`lib.removePrefix` was broken - it was never adjusted when porting this
from ofborg, so the relative path was wrong and no prefix ever removed,
since no packages are in `ci/`.

This additionally strips the leading `/`, so that `relevantFilenames`
will then have paths starting with `pkgs/...`, similar to how git
reports those paths in the `changedpathsjson` file. This allows simple
equality comparison.
2025-10-17 16:37:04 +02:00
Wolfgang Walther
79029e39a4 ci/eval/compare/maintainers: refactor using lib
Removes unused code and comments, uses shorter lib over builtins and
switches to `lib.importJSON`.

Readability is key.
2025-10-17 16:37:00 +02:00
Wolfgang Walther
44e4c1ba9f maintainers/scripts/get-maintainer-pings-between: drop
This has been broken for months, since all the interface to `ci/eval`
changed. Nobody noticed, so nobody is using it.
2025-10-16 13:36:42 +02:00
Vladimír Čunát
dd51f936da Merge branch 'master' into staging-next
lixPackageSets.git conflicted via 9c29a340be + eabfa160aa
I think, hopefully good now.
2025-10-05 08:18:28 +02:00
NAHO
dec908b169 treewide: remove optional builtins prefixes from prelude functions
Remove optional builtins prefixes from prelude functions by running:

    builtins=(
      abort
      baseNameOf
      break
      derivation
      derivationStrict
      dirOf
      false
      fetchGit
      fetchMercurial
      fetchTarball
      fetchTree
      fromTOML
      import
      isNull
      map
      null
      placeholder
      removeAttrs
      scopedImport
      throw
      toString
      true
    )

    fd --type file --exec-batch sed --in-place --regexp-extended "
      s/\<builtins\.($(
        printf '%s\n' "${builtins[@]}" |
          paste --delimiter '|' --serial -
      ))\>/\1/g
    "

    nix fmt
2025-09-30 09:05:08 +02:00
nixpkgs-ci[bot]
15409f9947 Merge staging-next into staging 2025-09-18 18:06:41 +00:00
Philip Taron
3d0e2841b8 ci.eval.compare: keep warnings as warnings rather than raising them as errors
This reverts commit f8210561f3 (ci.eval.compare: turn warnings into errors, 2025-09-16).

It turns out that there are normal math warnings and we don't want to block CI on the math coming out wrong.
2025-09-18 08:37:36 -07:00
Philip Taron
d80d4a77b7 ci.eval.compare: split out equivalent values into their own table 2025-09-17 14:04:28 -07:00
Philip Taron
fb1647ec6e ci.eval.compare: explain the various metrics under the --explain flag 2025-09-17 14:04:28 -07:00
Philip Taron
c9860ef95c ci.eval.compare: remove the duplicate cpuTime key 2025-09-17 14:04:28 -07:00
Philip Taron
2817f79649 ci.eval.compare: put things with counts together 2025-09-17 14:04:28 -07:00
Philip Taron
210e3e1151 ci.eval.compare: put things with bytes together 2025-09-17 14:04:28 -07:00
Philip Taron
3edc1e204e ci.eval.compare: make the table format using tabulate not manually 2025-09-17 14:04:28 -07:00
Philip Taron
4eaa094f20 ci.eval.compare: sort time metrics first, then GC metrics, then everything else 2025-09-17 14:04:28 -07:00
Philip Taron
e83e900874 ci.eval.compare: assert types in flatten_data 2025-09-17 14:04:28 -07:00
Philip Taron
9959a4e507 ci.eval.compare: delete unreferenced global 2025-09-17 14:04:28 -07:00
Philip Taron
e4101ea3a9 ci.eval.compare: instead of manually tabulating, use tabulate 2025-09-17 14:04:28 -07:00
Philip Taron
7818a245f7 ci.eval.compare: support passing single files to cmp-stats 2025-09-17 14:04:27 -07:00
Philip Taron
4bc54e7a3a ci.eval.compare: allow before_vals == 1 but avoid the t-test 2025-09-17 14:04:27 -07:00
Philip Taron
f8210561f3 ci.eval.compare: turn warnings into errors
This helps detect my math errors
2025-09-17 14:04:27 -07:00
Philip Taron
4c7ec9bf20 ci.eval.compare: require the directories to exist (they always should) 2025-09-17 14:04:27 -07:00
Philip Taron
241bb94b64 ci.eval.compare: use argument parsing instead of environment variables to pass before/after to cmp-stats.py 2025-09-17 14:04:27 -07:00
Philip Taron
2fe7b1cec2 ci.eval.compare: extract a derivation for cmp-stats
It's not very useful yet.
2025-09-17 14:04:27 -07:00
Philip Taron
4c2c6d9c43 ci.eval.compare: extract main function in cmp-stats.py 2025-09-17 14:04:27 -07:00
Philip Taron
6900cf62ee ci.eval.compare: sort imports in cmp-stats.py 2025-09-17 14:04:27 -07:00
Philip Taron
a3c01782d3 ci.eval.compare: format cmp-stats.py with ruff 2025-09-17 14:04:27 -07:00
nixpkgs-ci[bot]
6ec57b45c7 Merge staging-next into staging 2025-09-15 16:09:34 +00:00
Wolfgang Walther
e5ce5c2500 ci/eval/compare: add 10.rebuild-nixos-tests label (#439255) 2025-09-15 14:29:47 +00:00
nixpkgs-ci[bot]
96acd252c7 Merge staging-next into staging 2025-09-12 00:16:59 +00:00
Wolfgang Walther
b9d4098221 ci/eval: fix local full eval
The change to use `builtins.storePath` was good - for when the store
path *is* already part of the nix store. In all my tests so far, that
was already the case, because I was iterating on the solution and the
Eval results stayed the same.

But when this is run on a entirely new commit, these the values for
`afterDir` and `combinedDir` are *not* in the store, yet. As part of
running `eval.full` on a new commit they will be created. `eval.full` is
linked up, so that the values passed around there will actually be
derivations, which might not be realized, yet.

Checking whether the input is a path or not fixes this for both cases.
2025-09-11 20:18:48 +02:00
nixpkgs-ci[bot]
3e0c2daff8 Merge staging-next into staging 2025-09-10 18:06:27 +00:00
Wolfgang Walther
45a765a282 ci/eval: fix local comparison with baseline
Due to how we pass in existing store paths via CLI arguments for the
diff and combine scripts, Nix didn't register a dependency on the store
paths properly. This meant that some of the derivations that were built,
didn't have the right store paths made available in the sandbox -
leading to all kinds of "not found" errors.

We worked around this in CI by resolving the symlinks to the nix store
beforehand. We tried to work around this locally by storing the nix
store path in BASELINE, but this didn't fully work. By explicitly
registering these store paths as dependencies, this should work across
the board - without any magic required by the caller.
2025-09-10 14:35:46 +02:00
Wolfgang Walther
b9ec8bed14 treewide: fix typos in md files (#439306) 2025-09-09 08:03:31 +00:00
Ben Siraphob
807ce4b7b3 ci/eval/README.md: adjust wording 2025-09-09 13:14:36 +07:00
Philip Taron
07916fc3fd ci: have eval.full return the report as displayed in CI
Update the README.md to document what gets returned.
We might in the future split these up into other attrsets but I don't see a usecase for that at the moment.
2025-09-08 06:58:24 -07:00
Wolfgang Walther
63e6fc4c43 ci/eval/compare: add 10.rebuild-nixos-tests label
This indicates that the NixOS test-driver changed and all NixOS tests
have to be rebuilt. It can be used to either re-target to staging or to
batch this with other similar changes, at least.
2025-09-04 21:27:30 +02:00
Wolfgang Walther
e3f583c5bf ci/eval: eval nixosTests.simple
We can't eval all nixos tests, this would be way too expensive. But we
can evaluate `nixosTests.simple` as an indication whether the nixos test
driver has changed. If that's the case, this means that *all* NixOS
tests need to be rebuilt.
2025-09-03 18:35:01 +02:00
Wolfgang Walther
5c4608656e top-level/release-outpaths: move to ci/eval & refactor (#439221) 2025-09-03 16:27:37 +00:00
Wolfgang Walther
f2ca5796de ci/eval/compare: handle missing packages
Not all packages that are reported as changed will actually exist on the
platform that the maintainers are colleted on.

This is the case for some attributes that are only available on Darwin
or explicitly set to `null` on Linux. By filtering out packages without
maintainers, these are ignored - and we should potentially get a small
performance improvement as well.
2025-09-01 15:50:05 +02:00
Wolfgang Walther
8104100101 ci/eval/attrpaths: update cross stdenvs
`pkgsArocc` and `pkgsZig` had been added to `release.nix`, so should be
listed here as well.
2025-08-31 21:09:31 +02:00
Wolfgang Walther
d0c0b875f7 treewide: remove __recurseIntoDerivationForReleaseJobs
This attribute was supposed to be set on derivations, to make the
release tools recurse into them. The remaining uses were all on regular
attrsets, though, so this is safe to remove.
2025-08-31 20:54:49 +02:00
Wolfgang Walther
b627d181e9 ci/eval: remove unused checkMeta argument
This should always be set anyways.
2025-08-31 12:16:28 +02:00
Wolfgang Walther
04fcbb45e1 ci/eval/attrpaths: refactor
The following changes were made:
- Using `lib.` instead of `builtins.`
- Using `mapAttrsToList` instead of `mapAttrs` + `attrValues`
- Joining two of the if conditions with the same return value
- Using `traceIf` instead of `if` / `else`
- Using `showAttrPath` instead of `concatStringsSep`
2025-08-31 12:16:27 +02:00
Wolfgang Walther
9524a21fe0 ci/eval/attrpaths: remove left-over condition
This condition doesn't make a difference anymore, ever since we removed
the tryEval code from this file and had already enabled unfree packages
earlier anyway.
2025-08-31 12:15:11 +02:00
Wolfgang Walther
2aae142529 ci/eval: remove ofborg references
By now, these files have been changed enough to not need the "vendored
from" notes anymore. These links would still be there when going through
the history of the file, but today GHA CI has not many similarities
anymore to what ofborg did, so these are not really helpful.
2025-08-31 12:15:06 +02:00
Wolfgang Walther
0d51e920d3 top-level/release-outpaths: move to ci/eval
These files are tightly coupled with the code in ci/eval and not used
anywhere else. They are subject to the same backporting requirements as
the remaining CI code. They are better placed next here.
2025-08-31 12:15:02 +02:00
Wolfgang Walther
4126ef7e00 ci/eval/compare: refactor
Simplification after the last step.
2025-08-30 17:56:35 +02:00
Wolfgang Walther
e88dd3a8b2 ci/eval/compare: only check changed attrpaths
It makes no sense to check newly added attrpaths for maintainers on the
target branch - by definition these attrpaths won't exist, yet. We can
avoid falling back to `null` for these etc.
2025-08-30 17:56:30 +02:00