From 2e07c7808660e42ebfec4906ca9b76b88ed5e27b Mon Sep 17 00:00:00 2001 From: Wolfgang Walther Date: Sun, 29 Jun 2025 19:14:47 +0200 Subject: [PATCH 1/3] ci/README.md: one sentence per line (cherry picked from commit 99f2e655d9db009ee0b4ede3edced5f6c882c7f4) (cherry picked from commit 8d80a5164fbfee749e6ab5cf2c88133f17f6dee4) --- ci/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/ci/README.md b/ci/README.md index 7df3b1533e59..c55d0ca62d07 100644 --- a/ci/README.md +++ b/ci/README.md @@ -6,20 +6,21 @@ This is in contrast with [`maintainers/scripts`](../maintainers/scripts) which i ## Pinned Nixpkgs CI may need certain packages from Nixpkgs. -In order to ensure that the needed packages are generally available without building, -[`pinned.json`](./pinned.json) contains a pinned Nixpkgs version tested by Hydra. +In order to ensure that the needed packages are generally available without building, [`pinned.json`](./pinned.json) contains a pinned Nixpkgs version tested by Hydra. Run [`update-pinned.sh`](./update-pinned.sh) to update it. ## `ci/nixpkgs-vet.sh BASE_BRANCH [REPOSITORY]` -Runs the [`nixpkgs-vet` tool](https://github.com/NixOS/nixpkgs-vet) on the HEAD commit, closely matching what CI does. This can't do exactly the same as CI, because CI needs to rely on GitHub's server-side Git history to compute the mergeability of PRs before the check can be started. +Runs the [`nixpkgs-vet` tool](https://github.com/NixOS/nixpkgs-vet) on the HEAD commit, closely matching what CI does. +This can't do exactly the same as CI, because CI needs to rely on GitHub's server-side Git history to compute the mergeability of PRs before the check can be started. In turn, when contributors are running this tool locally, we don't want to have to push commits to test them, and we can also rely on the local Git history to do the mergeability check. Arguments: - `BASE_BRANCH`: The base branch to use, e.g. master or release-24.05 -- `REPOSITORY`: The repository from which to fetch the base branch. Defaults to . +- `REPOSITORY`: The repository from which to fetch the base branch. + Defaults to . # Branch classification From 11ccc438638e165b918d7a5f48748ec34086ba59 Mon Sep 17 00:00:00 2001 From: Wolfgang Walther Date: Sun, 29 Jun 2025 19:14:57 +0200 Subject: [PATCH 2/3] ci/eval/README.md: one sentence per line (cherry picked from commit 60e35e4ded6e91524364a74b3b4ec233ed9321f2) (cherry picked from commit 51f994d5b5301d4d55ebc025f6bdd8de4a939890) --- ci/eval/README.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/ci/eval/README.md b/ci/eval/README.md index 011f3dd74ed0..ec7429b7bc78 100644 --- a/ci/eval/README.md +++ b/ci/eval/README.md @@ -11,10 +11,15 @@ nix-build ci -A eval.full \ --arg evalSystems '["x86_64-linux" "aarch64-darwin"]' ``` -- `--max-jobs`: The maximum number of derivations to run at the same time. Only each [supported system](../supportedSystems.json) gets a separate derivation, so it doesn't make sense to set this higher than that number. -- `--cores`: The number of cores to use for each job. Recommended to set this to the amount of cores on your system divided by `--max-jobs`. -- `chunkSize`: The number of attributes that are evaluated simultaneously on a single core. Lowering this decreases memory usage at the cost of increased evaluation time. If this is too high, there won't be enough chunks to process them in parallel, and will also increase evaluation time. -- `evalSystems`: The set of systems for which `nixpkgs` should be evaluated. Defaults to the four official platforms (`x86_64-linux`, `aarch64-linux`, `x86_64-darwin` and `aarch64-darwin`). +- `--max-jobs`: The maximum number of derivations to run at the same time. + Only each [supported system](../supportedSystems.json) gets a separate derivation, so it doesn't make sense to set this higher than that number. +- `--cores`: The number of cores to use for each job. + Recommended to set this to the amount of cores on your system divided by `--max-jobs`. +- `chunkSize`: The number of attributes that are evaluated simultaneously on a single core. + Lowering this decreases memory usage at the cost of increased evaluation time. + If this is too high, there won't be enough chunks to process them in parallel, and will also increase evaluation time. +- `evalSystems`: The set of systems for which `nixpkgs` should be evaluated. + Defaults to the four official platforms (`x86_64-linux`, `aarch64-linux`, `x86_64-darwin` and `aarch64-darwin`). A good default is to set `chunkSize` to 10000, which leads to about 3.6GB max memory usage per core, so suitable for fully utilising machines with 4 cores and 16GB memory, 8 cores and 32GB memory or 16 cores and 64GB memory. From 3bcaebc6035750592b759f2ad3e8c009ba074520 Mon Sep 17 00:00:00 2001 From: Wolfgang Walther Date: Sun, 29 Jun 2025 19:15:06 +0200 Subject: [PATCH 3/3] .github/workflows/README.md: one sentence per line (cherry picked from commit 3d505c03610b6102af6d870ae3506a151cef1f68) (cherry picked from commit e80b7744ac9111f9c5af1bdf0bc7a695c8489646) --- .github/workflows/README.md | 47 +++++++++++++++++++++++++++---------- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index e3be33ae6456..c2a16827ebd4 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -2,21 +2,35 @@ Some architectural notes about key decisions and concepts in our workflows: -- Instead of `pull_request` we use [`pull_request_target`](https://docs.github.com/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target) for all PR-related workflows. This has the advantage that those workflows will run without prior approval for external contributors. +- Instead of `pull_request` we use [`pull_request_target`](https://docs.github.com/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target) for all PR-related workflows. + This has the advantage that those workflows will run without prior approval for external contributors. -- Running on `pull_request_target` also optionally provides us with a GH_TOKEN with elevated privileges (write access), which we need to do things like adding labels, requesting reviewers or pushing branches. **Note about security:** We need to be careful to limit the scope of elevated privileges as much as possible. Thus they should be lowered to the minimum with `permissions: {}` in every workflow by default. +- Running on `pull_request_target` also optionally provides us with a GH_TOKEN with elevated privileges (write access), which we need to do things like adding labels, requesting reviewers or pushing branches. + **Note about security:** We need to be careful to limit the scope of elevated privileges as much as possible. + Thus they should be lowered to the minimum with `permissions: {}` in every workflow by default. -- By definition `pull_request_target` runs in the context of the **base** of the pull request. This means, that the workflow files to run will be taken from the base branch, not the PR, and actions/checkout will not checkout the PR, but the base branch, by default. To protect our secrets, we need to make sure to **never execute code** from the pull request and always evaluate or build nix code from the pull request with the **sandbox enabled**. +- By definition `pull_request_target` runs in the context of the **base** of the pull request. + This means, that the workflow files to run will be taken from the base branch, not the PR, and actions/checkout will not checkout the PR, but the base branch, by default. + To protect our secrets, we need to make sure to **never execute code** from the pull request and always evaluate or build nix code from the pull request with the **sandbox enabled**. -- To test the pull request's contents, we checkout the "test merge commit". This is a temporary commit that GitHub creates automatically as "what would happen, if this PR was merged into the base branch now?". The checkout could be done via the virtual branch `refs/pull//merge`, but doing so would cause failures when this virtual branch doesn't exist (anymore). This can happen when the PR has conflicts, in which case the virtual branch is not created, or when the PR is getting merged while workflows are still running, in which case the branch won't exist anymore at the time of checkout. Thus, we use the `get-merge-commit.yml` workflow to check whether the PR is mergeable and the test merge commit exists and only then run the relevant jobs. +- To test the pull request's contents, we checkout the "test merge commit". + This is a temporary commit that GitHub creates automatically as "what would happen, if this PR was merged into the base branch now?". + The checkout could be done via the virtual branch `refs/pull//merge`, but doing so would cause failures when this virtual branch doesn't exist (anymore). + This can happen when the PR has conflicts, in which case the virtual branch is not created, or when the PR is getting merged while workflows are still running, in which case the branch won't exist anymore at the time of checkout. + Thus, we use the `get-merge-commit.yml` workflow to check whether the PR is mergeable and the test merge commit exists and only then run the relevant jobs. -- Various workflows need to make comparisons against the base branch. In this case, we checkout the parent of the "test merge commit" for best results. Note, that this is not necessarily the same as the default commit that actions/checkout would use, which is also a commit from the base branch (see above), but might be older. +- Various workflows need to make comparisons against the base branch. + In this case, we checkout the parent of the "test merge commit" for best results. + Note, that this is not necessarily the same as the default commit that actions/checkout would use, which is also a commit from the base branch (see above), but might be older. ## Terminology -- **base commit**: The pull_request_target event's context commit, i.e. the base commit given by GitHub Actions. Same as `github.event.pull_request.base.sha`. -- **head commit**: The HEAD commit in the pull request's branch. Same as `github.event.pull_request.head.sha`. -- **merge commit**: The temporary "test merge commit" that GitHub Actions creates and updates for the pull request. Same as `refs/pull/${{ github.event.pull_request.number }}/merge`. +- **base commit**: The pull_request_target event's context commit, i.e. the base commit given by GitHub Actions. + Same as `github.event.pull_request.base.sha`. +- **head commit**: The HEAD commit in the pull request's branch. + Same as `github.event.pull_request.head.sha`. +- **merge commit**: The temporary "test merge commit" that GitHub Actions creates and updates for the pull request. + Same as `refs/pull/${{ github.event.pull_request.number }}/merge`. - **target commit**: The base branch's parent of the "test merge commit" to compare against. ## Concurrency Groups @@ -25,15 +39,22 @@ We use [GitHub's Concurrency Groups](https://docs.github.com/en/actions/writing- When two workflows are in the same group, a newer workflow cancels an older workflow. Thus, it is important how to construct the group keys: -- Because we want to run jobs for different events at same time, we add `github.event_name` to the key. This is the case for the `pull_request` which runs on changes to the workflow files to test the new files and the same workflow from the base branch run via `pull_request_event`. +- Because we want to run jobs for different events at same time, we add `github.event_name` to the key. + This is the case for the `pull_request` which runs on changes to the workflow files to test the new files and the same workflow from the base branch run via `pull_request_event`. -- We don't want workflows of different Pull Requests to cancel each other, so we include `github.event.pull_request.number`. The [GitHub docs](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/control-the-concurrency-of-workflows-and-jobs#example-using-a-fallback-value) show using `github.head_ref` for this purpose, but this doesn't work well with forks: Different users could have the same head branch name in their forks and run CI for their PRs at the same time. +- We don't want workflows of different Pull Requests to cancel each other, so we include `github.event.pull_request.number`. + The [GitHub docs](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/control-the-concurrency-of-workflows-and-jobs#example-using-a-fallback-value) show using `github.head_ref` for this purpose, but this doesn't work well with forks: Different users could have the same head branch name in their forks and run CI for their PRs at the same time. -- Sometimes, there is no `pull_request.number`. To ensure non-PR runs are never cancelled, we add a fallback of `github.run_id`. This is a unique value for each workflow run. +- Sometimes, there is no `pull_request.number`. + To ensure non-PR runs are never cancelled, we add a fallback of `github.run_id`. + This is a unique value for each workflow run. -- Of course, we run multiple workflows at the same time, so we add `github.workflow` to the key. Otherwise workflows would cancel each other. +- Of course, we run multiple workflows at the same time, so we add `github.workflow` to the key. + Otherwise workflows would cancel each other. -- There is a special case for reusable workflows called via `workflow_call` - they will have `github.workflow` set to their parent workflow's name. Thus, they would cancel each other. That's why we additionally hardcode the name of the workflow as well. +- There is a special case for reusable workflows called via `workflow_call` - they will have `github.workflow` set to their parent workflow's name. + Thus, they would cancel each other. + That's why we additionally hardcode the name of the workflow as well. This results in a key with the following semantics: