Clean up leftover for commit cd13136f03
("fetchurl: use __structuredAttrs = true and pass curlOptsList directly")
Continue the work of commit 23236b331d
("fetchurl: fix handling of fallback URLs"),
addressing a Bash array re-assignment quirk:
when assigning a Bash array variable as if it were a plain variable,
the value goes to the first element,
and the rest of the array stays the same.
```console
$ foo=(a b)
$ declare -p foo
declare -a foo=([0]="a" [1]="b")
$ foo="c d"
$ declare -p foo
declare -a foo=([0]="c d" [1]="b")
```
Don't rewrite the `${urls[@]}` with resolved URLs,
but hold them with `${resolvedUrls[@]}` instead.
Co-authored-by: Matt Sturgeon <matt@sturgeon.me.uk>
Co-authored-by: Wolfgang Walther <walther@technowledgy.de>
Without the change fallback to `urls = []` does not
work as `$urls` evaluates on ony first entry. Use
`${urls[@]}` instead.
Example derivation that is fixed in `master` is
`xterm.src`:
Before:
$ nix build --no-link -f. xterm.src --rebuild -L
xterm> structuredAttrs is enabled
...
xterm> trying ftp://ftp.invisible-island.net/xterm/xterm-403.tgz
...
xterm> curl: (67) Access denied: 550
xterm> error: cannot download xterm-403.tgz from any mirror
After:
$ nix build --no-link -f. xterm.src --rebuild -L
xterm> structuredAttrs is enabled
...
xterm> trying ftp://ftp.invisible-island.net/xterm/xterm-403.tgz
...
xterm> curl: (67) Access denied: 550
...
xterm> trying https://invisible-mirror.net/archives/xterm/xterm-403.tgz
xterm> % Total % Received % Xferd Average Speed Time Time Time Current
xterm> Dload Upload Total Spent Left Speed
xterm> 100 1577k 100 1577k 0 0 703670 0 0:00:02 0:00:02 --:--:-- 703866
fetchurl supports postFetch, which must not run when fetching from
hashedMirrors, because they provide pre-built derivation outputs.
fetchurl also supports downloadToTemp, which modifies behavior relating
to postFetch.
Before this commit, a call like
fetchurl {
url = "...";
downloadToTemp = true;
postFetch = ''
mv "$downloadedFile" "$out"
'';
}
would work in normal scenarios, but fail when fetching from
hashedMirrors, due to how internally downloadToTemp was not tied to
postFetch as closely as it should've been. This resulted in an error
like
builder [...] failed to produce output path
This commit fixes this, making hashedMirror fetching work even for
derivations that have downloadToTemp enabled.
Co-Authored-By: Silvan Mosberger <silvan.mosberger@tweag.io>
It's hard to put the sourcing of ./.attrs.sh into all builder
consistently - mistakes will happen. Thus, load structured attrs once in
make-derivation and then source the remaining builder on top.
This should fix quite a few builders with structured attributes in
principle. Most importantly it helps substitute / substituteAll, which
are required for bootstrap on some platforms.
current invocation is extremely fragile if tcp sockets are spuriously shutdown (this can happen when changing networks, or just due to a bad ISP)
--retry-all-errors is the only way to make curl robust against this type of failure.
This is a follow-up to a169553f7e.
In most cases it should allow the TLS verification to be enabled.
It also makes the behavior of `fetchurl` more consistent with other fetchers
like `fetchgit`.
Ideally we would always fallback on `cacert` but I am not sure how to build
`cacert` during bootstrap without making an unmaintainable mess.
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into
builtins.derivation {
builder = stdenv.shell;
args = [ "-e" builder ];
}
This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].
Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:
* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
(see previous commit for more context), so prefer the environment
variable if possible.
* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
can happen.
* If neither applies, it can be assumed that `__structuredAttrs` is
turned off and thus nothing needs to be done.
[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
it can point to a wrong location on older Nix versions with a bug in
`__structuredAttrs`.
It's impossible to pass arguments with spaces with curlOpts.
curlOptsList supports that. Passing a list to curlOpts has been
deprecated. This commit is fully backwards compatible.
We can use cacert to validate that the data passes SSL certificates.
Normally, this doesn’t happen because we already have the hash, but in
the hash = "" case we don’t.
hashed-mirrors are content addressed. So if $outputHash is in the
hashed-mirror, changes from ‘postFetch’ would already be made. So,
running postFetch will end up applying the change /again/, which we
don’t want.
It would be nice to be able to track Nix requests. It's not trustworthy,
but can be helpful for stats and routing in HTTP logs.
Since `fetchurl` is used so widely, we should "magically" get a UA on
`fetchzip`, `fetchFromGitHub`, and other related fetchers.
Since `fetchurl` is only used for fixed-output derivations, this should
cause no mass rebuild.
User-Agent example: curl/7.57.0 Nixpkgs/18.03
Fixes#27406.
Commit 5d4efb2c81 added an assertion to `stopNest'
which requires it be correctly paired with `startNest'. `fetchurl' calls
`stopNest', but never calls `startNest'; the former calls are removed.
Otherwise, if the upstream mirror changes (rather than deletes) a
file, then tarballs.nixos.org won't be used even if it has a copy of
the original file, and so we'll get a hash mismatch.
This function downloads and unpacks a file in one fixed-output
derivation. This is primarily useful for dynamically generated zip
files, such as GitHub's /archive URLs, where the unpacked content of
the zip file doesn't change, but the zip file itself may (e.g. due to
minor changes in the compression algorithm, or changes in timestamps).
Fetchzip is implemented by extending fetchurl with a "postFetch" hook
that is executed after the file has been downloaded. This hook can
thus perform arbitrary checks or transformations on the downloaded
file.
fetchurl instantiations, instead of passing the mirrors to fetchurl
instantiations via environment variables. This makes the resulting
store derivations (.drv files) much smaller, which in turn makes
nix-env/nix-instantiate faster (4.8 -> 4.2 seconds on nix-env -qa
--out-path).
svn path=/nixpkgs/trunk/; revision=12695
mirror:// sites through environment variables, e.g.
NIX_MIRRORS_gnu="ftp://ftp.nluug.nl/pub/gnu/ ftp://ftp.gnu.org/pub/gnu/"
or
NIX_MIRRORS_sourceforge="http://surfnet.dl.sourceforge.net/sourceforge/"
svn path=/nixpkgs/trunk/; revision=9302
fetchurl {
url = http://heanet.dl.sourceforge.net/sourceforge/zapping/zapping-0.9.6.tar.bz2;
md5 = "8306775c6a11de4d72345b5eee970ea6";
};
you can write
fetchurl {
url = mirror://sourceforge/zapping/zapping-0.9.6.tar.bz2;
md5 = "8306775c6a11de4d72345b5eee970ea6";
};
which causes fetchurl to try the SourceForge mirrors listed in the
`sourceforge' attribute in build-support/fetchurl/mirrors.nix.
(They're currently tried in sequence, and the lists of mirrors are
not configurable yet.)
The syntax for mirror URLs is mirror://site/path/to/file, where
`site' is currently one of `sourceforge', `gnu' (mirrors of
ftp://ftp.gnu.org/pub/gnu) and `kernel' (mirrors of
http://www.all.kernel.org/pub/).
svn path=/nixpkgs/trunk/; revision=9197
fetching a file with hash HASH of type TYPE, we first try to
download <base-url>/<type>/<hash>, where <base-url> is one of a list
of mirrors. For instance, given
src = fetchurl {
url = http://releases.mozilla.org/pub/mozilla.org/firefox/releases/2.0.0.6/source/firefox-2.0.0.6-source.tar.bz2;
sha1 = "eb72f55e4a8bf08e8c6ef227c0ade3d068ba1082";
};
and the mirror list [http://nix.cs.uu.nl/dist/tarballs], we first
try to download
eb72f55e4a
and if that fails, we use the original URL.
The list of mirrors is not yet user-configurable.
* `fetchurl' now also accepts an argument `urls' instead of `url' for
a list of alternative download locations, which fetchurl will try in
sequence.
svn path=/nixpkgs/trunk/; revision=9190
