59d55cbaa3
Adds a source filtering function inspired by [doublestar](https://github.com/bmatcuk/doublestar).
This has been in used in a few private repositories since the last ~6 months with success.
- Testing
This was originally tested with the nix-unit testsuite:
```
let
inherit (import ./internal.nix) mkSourceFilter mkMatcher;
in
{
mkMatcher = {
empty = {
testMatch = {
expr = mkMatcher "" "" "regular";
expected = true;
};
testNoMatch = {
expr = mkMatcher "" "foo" "regular";
expected = false;
};
};
simple = {
testMatch = {
expr = mkMatcher "foo" "foo" "regular";
expected = true;
};
testNoMatch = {
expr = mkMatcher "foo" "bar" "regular";
expected = false;
};
};
singleStar = {
testMatch = {
expr = mkMatcher "*.js" "foo.js" "regular";
expected = true;
};
testNoMatch = {
expr = mkMatcher "*.js" "foo.py" "regular";
expected = false;
};
};
doubleStar = {
testMatch = {
expr = mkMatcher "foo/**/bar" "foo/baz/bar" "regular";
expected = true;
};
testNoMatch = {
expr = mkMatcher "foo/**/bar" "foo/bar/baz" "regular";
expected = false;
};
testMultiMatch = {
expr = mkMatcher "foo/**/bar" "foo/baz/xyz/bar" "regular";
expected = true;
};
testMultiMatchDoubleGlob = {
expr = mkMatcher "foo/**/**/bar" "foo/baz/xyz/bar" "regular";
expected = true;
};
testInfixMatch = {
expr = mkMatcher "foo/**/qux/**/bar" "foo/baz/qux/baz/bar" "regular";
expected = true;
};
testInfixNoMatch = {
expr = mkMatcher "foo/**/xyz/**/bar" "foo/baz/qux/baz/bar" "regular";
expected = false;
};
# Technically a partial match
testInfixDirMatch = {
expr = mkMatcher "foo/**/xyz/**/bar" "foo/baz/qux/baz/bar" "directory";
expected = true;
};
};
};
mkSourceFilter = {
testSourceFilter = {
expr = mkSourceFilter ./fixtures [
"bar/*.js"
] "bar/bar.js" "regular";
expected = true;
};
};
}
```
but it was dropped in this nixpkgs contribution as the structure of nixpkgs lib testing is too primitive to incorp this without more extensive refactoring than I'd like at the momment.
- Performance
It's hard to benchmark this against anything else meaningful except [globsset](https://github.com/pdtpartners/globset), which has a very similar API.
`sourceByGlobs` avoids performance pitfalls by:
- Using `builtins.filterSource`
This is more performant than the fileset API.
The downside compared to the fileset API is that any directory which matches the filter will be added to the build, even if it's empty.
- Match paths component by component
By splitting each pattern into a token per / separator.
This is much faster in Nix than the doublestar algorithm.
- Globset source
```json
{
"cpuTime": 0.8585879802703857,
"envs": {
"bytes": 148252864,
"elements": 11899843,
"number": 6631765
},
"gc": {
"heapSize": 402915328,
"totalBytes": 671288560
},
"list": {
"bytes": 3358664,
"concats": 28658,
"elements": 419833
},
"nrAvoided": 11562713,
"nrFunctionCalls": 4816963,
"nrLookups": 4316209,
"nrOpUpdateValuesCopied": 5686407,
"nrOpUpdates": 464060,
"nrPrimOpCalls": 2966970,
"nrThunks": 7796186,
"sets": {
"bytes": 196404672,
"elements": 10837802,
"number": 1437490
},
"sizes": {
"Attr": 16,
"Bindings": 16,
"Env": 8,
"Value": 24
},
"symbols": {
"bytes": 340652,
"number": 32026
},
"values": {
"bytes": 207367440,
"number": 8640310
}
}
```
- Glob-filter source
```json
{
"cpuTime": 0.3904629945755005,
"envs": {
"bytes": 13263440,
"elements": 1005877,
"number": 652053
},
"gc": {
"heapSize": 402915328,
"totalBytes": 146914896
},
"list": {
"bytes": 3032168,
"concats": 5899,
"elements": 379021
},
"nrAvoided": 1666598,
"nrFunctionCalls": 484399,
"nrLookups": 112698,
"nrOpUpdateValuesCopied": 3432135,
"nrOpUpdates": 13426,
"nrPrimOpCalls": 1041954,
"nrThunks": 1205792,
"sets": {
"bytes": 64304800,
"elements": 3978167,
"number": 40883
},
"sizes": {
"Attr": 16,
"Bindings": 16,
"Env": 8,
"Value": 24
},
"symbols": {
"bytes": 285306,
"number": 28864
},
"values": {
"bytes": 42963240,
"number": 1790135
}
}
```
Nixpkgs is a collection of over 120,000 software packages that can be installed with the Nix package manager. It also implements NixOS, a purely-functional Linux distribution.
Manuals
- NixOS Manual - how to install, configure, and maintain a purely-functional Linux distribution
- Nixpkgs Manual - contributing to Nixpkgs and using programming-language-specific Nix expressions
- Nix Package Manager Manual - how to write Nix expressions (programs), and how to use Nix command line tools
Community
- Discourse Forum
- Matrix Chat
- Official wiki
- Community-maintained list of ways to get in touch (Discord, Telegram, IRC, etc.)
Other Project Repositories
The sources of all official Nix-related projects are in the NixOS organization on GitHub. Here are some of the main ones:
- Nix - the purely functional package manager
- NixOps - the tool to remotely deploy NixOS machines
- nixos-hardware - NixOS profiles to optimize settings for different hardware
- Nix RFCs - the formal process for making substantial changes to the community
- NixOS homepage - the NixOS.org website
- hydra - our continuous integration system
- NixOS Branding - NixOS branding
Continuous Integration and Distribution
Nixpkgs and NixOS are built and tested by our continuous integration system, Hydra.
- Continuous package builds for unstable/master
- Continuous package builds for the NixOS 25.11 release
- Tests for unstable/master
- Tests for the NixOS 25.11 release
Artifacts successfully built with Hydra are published to cache at https://cache.nixos.org/. When successful build and test criteria are met, the Nixpkgs expressions are distributed via Nix channels.
Contributing
Nixpkgs is among the most active projects on GitHub. While thousands of open issues and pull requests might seem like a lot at first, it helps to consider it in the context of the scope of the project. Nixpkgs describes how to build tens of thousands of pieces of software and implements a Linux distribution. The GitHub Insights page gives a sense of the project activity.
Community contributions are always welcome through GitHub Issues and Pull Requests.
For more information about contributing to the project, please visit the contributing page.
Donations
The infrastructure for NixOS and related projects is maintained by a nonprofit organization, the NixOS Foundation. To ensure the continuity and expansion of the NixOS infrastructure, we are looking for donations to our organization.
You can donate to the NixOS Foundation through SEPA bank transfers or by using Open Collective:
License
Nixpkgs is licensed under the MIT License.
Note
MIT license does not apply to the packages built by Nixpkgs, merely to the files in this repository (the Nix expressions, build scripts, NixOS modules, etc.). It also might not apply to patches included in Nixpkgs, which may be derivative works of the packages to which they apply. The aforementioned artifacts are all covered by the licenses of the respective packages.