The change to use `builtins.storePath` was good - for when the store
path *is* already part of the nix store. In all my tests so far, that
was already the case, because I was iterating on the solution and the
Eval results stayed the same.
But when this is run on a entirely new commit, these the values for
`afterDir` and `combinedDir` are *not* in the store, yet. As part of
running `eval.full` on a new commit they will be created. `eval.full` is
linked up, so that the values passed around there will actually be
derivations, which might not be realized, yet.
Checking whether the input is a path or not fixes this for both cases.
(cherry picked from commit b9d4098221)
Due to how we pass in existing store paths via CLI arguments for the
diff and combine scripts, Nix didn't register a dependency on the store
paths properly. This meant that some of the derivations that were built,
didn't have the right store paths made available in the sandbox -
leading to all kinds of "not found" errors.
We worked around this in CI by resolving the symlinks to the nix store
beforehand. We tried to work around this locally by storing the nix
store path in BASELINE, but this didn't fully work. By explicitly
registering these store paths as dependencies, this should work across
the board - without any magic required by the caller.
(cherry picked from commit 45a765a282)
Update the README.md to document what gets returned.
We might in the future split these up into other attrsets but I don't see a usecase for that at the moment.
(cherry picked from commit 07916fc3fd)
This attribute was supposed to be set on derivations, to make the
release tools recurse into them. The remaining uses were all on regular
attrsets, though, so this is safe to remove.
(cherry picked from commit d0c0b875f7)
The following changes were made:
- Using `lib.` instead of `builtins.`
- Using `mapAttrsToList` instead of `mapAttrs` + `attrValues`
- Joining two of the if conditions with the same return value
- Using `traceIf` instead of `if` / `else`
- Using `showAttrPath` instead of `concatStringsSep`
(cherry picked from commit 04fcbb45e1)
This condition doesn't make a difference anymore, ever since we removed
the tryEval code from this file and had already enabled unfree packages
earlier anyway.
(cherry picked from commit 9524a21fe0)
By now, these files have been changed enough to not need the "vendored
from" notes anymore. These links would still be there when going through
the history of the file, but today GHA CI has not many similarities
anymore to what ofborg did, so these are not really helpful.
(cherry picked from commit 2aae142529)
These files are tightly coupled with the code in ci/eval and not used
anywhere else. They are subject to the same backporting requirements as
the remaining CI code. They are better placed next here.
(cherry picked from commit 0d51e920d3)
Not all packages that are reported as changed will actually exist on the
platform that the maintainers are colleted on.
This is the case for some attributes that are only available on Darwin
or explicitly set to `null` on Linux. By filtering out packages without
maintainers, these are ignored - and we should potentially get a small
performance improvement as well.
(cherry picked from commit f2ca5796de)
It makes no sense to check newly added attrpaths for maintainers on the
target branch - by definition these attrpaths won't exist, yet. We can
avoid falling back to `null` for these etc.
(cherry picked from commit e88dd3a8b2)
This should not be necessary anymore, because packages that fail to
evaluate should already be filtered out by the attrpath generation step
in main eval.
(cherry picked from commit 0753aa4580)
This change pings maintainers of actually removed packages, aka where
the package's expression is deleted.
This will not ping maintainers of packages that become invisible,
because a (transitive) dependency of them is marked as insecure or
broken.
(cherry picked from commit 540e188796)
This allows running a full comparison between two commits locally.
What was previously `eval.full` is now called `eval.all`. The new
`eval.full` takes a `baseline` argument for the comparison.
(cherry picked from commit ccc12c839b)
We had set a default of 5000 for local evaluation earlier for
`singleSystem`, it makes sense to also use that for `full`.
The README is also a bit outdated, because Nix 2.30 significantly
changed the memory requirements. Rewriting the README to also show the
ability to directly evaluate the current system only.
(cherry picked from commit 0e07097947)
Instead of uploading the outpaths as artifact, this uploads them via
cachix. Most of all, this makes CI less brittle, because Eval in PRs
will still be able to succeed, even if no workflow run for the push
event could be found on the target branch. It will just take longer.
This also makes moving Eval into the Merge Queue easier to do: When
downloading artifacts from a different run, these would always have to
match on the right event, too. By pulling from cachix, the same workflow
can support target branches with merge queue and without merge queue at
the same time. The latter would still use the push event, while the
former could use the merge_group event.
Last but not least, this should fix Eval on PRs targeting `wip-`
branches and any other branches that the push event doesn't trigger on.
These would never find an Eval result from the target branch and could
never show rebuilds accurately. Now these PRs should work at a slightly
higher runtime cost.
(cherry picked from commit c1b06db57b)
Reason: keep ci directory in sync
- https://github.com/NixOS/nixpkgs/pull/431450#issuecomment-3209546418
This requires that we have a modules directory, in which case the
easy and robust solution is to only port the addition parts of the refactor.
It's about as easy as a .keep file, but more useful.
This means that some duplication is created, but we avoid backporting the
changes to the documentation generation, which is a somewhat complex
component I'd rather not touch until these changes have been proven out
on unstable.
Move "Packages" up, because it's much shorter and easier to scroll past.
This way both Packages and Performance are visible immediately.
(cherry picked from commit f05895fb3c)
Using these defaults makes it possible to just run `nix-build ci -A
eval.singleSystem` without passing additional arguments and get a sane
result back. Especially helpful when testing or debugging.
A `chunkSize` of 5000 is conservative to be able to run on systems with
less memory as well. Run-time is not impacted by that, as recent
benchmarks show.
(cherry picked from commit 75f40a150c)
`procps` pulls in 180 MB of systemd, but busybox also provides `kill`.
`busybox` also ships `time`, so no need for that extra dependency.
Using `nativeBuildInputs` pulls in all the -dev outputs of the listed
packages - which we don't need. We only need to run these tools, thus
map to their bin outputs.
Brings down the closure size from 500+ MB to 193 MB for the Eval job.
This probably saves ~10 seconds for the job.
(cherry picked from commit 5c697b8aea)
No need to return bytes in these files. Also busybox has problems to
render `free -b` with sizes > 100 GB properly in the next commit,
leading to extraction errors with awk.
(cherry picked from commit cb527a04e0)
When a package's attrpath is renamed it is currently treated as a
rebuild, even though the outpath already exists and is already cached.
This also happens when adding new names for packagesets that already
exist, for example when starting to eval `perlPackages` in CI, which is
just the same as `perl540Packages` currently. It would also happen when
`perlPackages` is switched from `perl540Packages` to `perl999Packages`.
Assuming that `perl999Packages` had already been built before, this
doesn't really cause any rebuilds.
(cherry picked from commit c5c7cd5bc7)
This file takes attrpaths as input and returns chunks of derivations,
aka a subset of release-outpaths - while release-attrpaths-superset
returns only an attrpath, not any derivations.
Thus, as a specialization of release-outpaths, the name
release-outpaths-parallel matches much better.
(cherry picked from commit 4ad7d21c88)
This doesn't fail on *all* asserts, yet, because nix-env still ignores
these in the main eval step. But it already gives some converage during
the attrpath generation.
(cherry picked from commit 30f19cce69)
Instead of deleting each label separately and then making another call
to add new labels, this replaces all labels at once, thus saving API
calls in some cases. Also, the labels are now managed in object-style
compared to the array-style before. This allows putting all the
knowledge about each label into a single place instead of in multiple
places. For example, the rebuild labels had to be special cased in the
workflow before - and the nix code to compare had to match that. Also,
the approval labels had to be considered in the `before` and `after`
phases.
The next commit shows how easy it is to add a new label now.
(cherry picked from commit 5b5b18c875)
Those have not been working since before the migration from OfBorg.
Those `rebuildsByKernel` are an attrset of lists coming from
`groupByKernel` (also see lengthy comment at the top of the file) - thus
we need `lib.elem` instead.
(cherry picked from commit 19ce5d94bc)
Previously we were taking nixVersions and this made external use from
the Lix repo's CI annoying.
We should probably also test other nix versions than stable (i.e. also
latest and Lix), but this involves writing GitHub Actions about it and
maybe not running it on every single PR. Future work.
(cherry picked from commit 332bc64369)
After the stats were added with a header, the list of packages now
appeared directly below it, without any separation.
(cherry picked from commit 4b31cabd6c)
We really can't expect packages that are marked as broken to evaluate,
and *especially* not on unsupported platforms.
For context, we were attempting to eval them *past* the broken throw
previously, which caused fun side effects like [0].
When we set `includeBroken = true` before, this also included unfree
packages. Those would now be excluded, which is not what we want. Thus,
we explicitly enable them separately.
Commit by winterqt, message slightly reworded by wolfgangwalther.
[0]:
https://github.com/NixOS/nixpkgs/issues/355847#issuecomment-2878873137
(cherry picked from commit 5240bdf3c6)
This moves the diff of outpaths into the outpaths job, mainly as a
preparation to allow future improvements. For example, this will allow
running the purity release checks only on changed outpaths instead of
the whole eval.
This also removes the inefficiency introduced in the last commit about
uploading the intermediate paths twice. Now, only the diff is passed on.
Also, technically, the diff is now run in parallel across 4 jobs. This
should be *slightly* faster than before, where outpaths from all systems
were combined first and then diffed. It's probably only a few seconds,
though.
(cherry picked from commit 8a39ce4a48)
Everything is a result, especially when nix-build uses "result" as its
default output. This becomes confusing, when re-wiring the different
parts later.
Thus, consistently name those things after some of their properties and
avoid the term result.
(cherry picked from commit b2579d36ff)
Instead of parsing a plain text file with jq, we can make nix-env output
JSON directly, which is significantly faster.
This saves about 8 out of 10 seconds for the combine step.
(cherry picked from commit 138393e0ba)
compare/maintainers.nix needs to access the current checkout to check
attrpaths, but makes the mistake of using lib from that checkout as
well. All other code in ci/ uses the pinned nixpkgs instance, so
maintainers.nix should do so as well.
(cherry picked from commit d87d760dfa)
Previously, `eval.full` organized the results for the supported systems
in a specific layout, i.e. with a folder with one subfolder per system.
Then, `eval.combine` relied on that.
When using `eval.singleSystem` and `eval.combine` directly, the caller
was responsible to recreate the same layout. This is annoying and
error-prone to do, when downloading artifacts from CI to recreate some
steps locally.
With this change, all the artifacts can be downloaded and extracted into
the same folder - because the result from `eval.singleSystem` already
contains the <system-name>/ subfolder.
(cherry picked from commit eccd9564ab)
This seems to be a left-over from before the performance comparison was
changed to a difference-per-chunk analysis.
(cherry picked from commit edaf51cb83)
While OfBorg is still adding these, it takes a much longer time to do so
compared to the eval action. Since we're adding rebuild labels, I think
it'd be nice to just do it within the eval action.
(cherry picked from commit d52066e2b1)
Right now, there are some paths that don't even get exposed to certain
systems (notably Darwin, but some outliers exist for Linux such as the
Darwin-specific Hackage overlay) for one reason or another, usually
because of assertions like `stdenv.isLinux`. To catch these scenarios,
this change implements a way to specify the system to evaluate attrpaths
on, and makes it default to the system that we're evaluating outpaths
for.
(cherry picked from commit 6b18b3d67b)
Previously, the attrs step consisted of:
- 7s queue time
- 1m 15s run time
Only 25s of this were spent preparing the attr paths. A bit more than a
minute was just spent for queuing, checking out the repo, downloading
nix, downloading dependencies, uploading the artifacts - and then
downloading them again in the next step. All of that can be avoided if
we collect the attrs as part of the outpaths job.
By running the attrs step as part of each outpaths step the attrpaths
will be collected 4x, but:
- We save a minute for each eval run to complete.
- We save a full job, giving us more free runners and *possibly* less
queue times for other jobs in the repo.
- We reduce complexity in the workflow file.
