`remarshal` is a Python application; evaluating its derivation forces a
large slice of `python3Packages` (rich, ruamel-yaml, cbor2, tomlkit and
their transitive build/check inputs — ~150 distinct python-modules,
~2700 derivations). 95 NixOS modules use `formats.toml`, so any system
enabling one of those services pays this on every `nixos-rebuild`.
`yj` is a small Go binary that does the same JSON→TOML conversion. For
all values accepted by `(formats.toml{}).type` the output is
semantically identical (verified via `tomllib`).
Reproducer (best of 3, `NIX_SHOW_STATS=1 nix-instantiate -E
'(pkgs.formats.toml{}).generate "x" {a.b=1;}'`):
cpuTime nrFunctionCalls nrThunks drv-closure
before (remarshal) 0.69s 634,117 1,184,499 2722 paths
after (yj) 0.36s 271,308 566,981 1481 paths
-49% -57% -52% -46%
Runtime closure of the converter: remarshal 202.3 MiB → yj 5.2 MiB.
Differences in output:
- yj emits intermediate table headers for deeply-nested keys (`[a]`
`[a.b]` `[a.b.c]` rather than just `[a.b.c]`). Valid TOML, parses
identically.
- yj renders very small/large floats as decimals rather than scientific
notation (`1e-300` → 300-digit literal). Valid TOML, parses
identically.
- Direct callers of `generate` that pass `null` (which the format type
already rejects) get key-dropped instead of a build-time error.
NixOS modules are unaffected since the option type filters this at
eval time; several modules already `filterAttrs (v: v != null)` for
exactly this reason.
Tested:
- `nix-build pkgs/pkgs-lib/tests -A formats` passes
- arrays of tables, nested tables, string escaping, int64, unicode
keys/values, mixed scalar+table siblings round-trip through `tomllib`
identically to remarshal output
adds `nixConf` (to generate `nix.conf` files) as a file format to
`lib.formats`, and uses it for serializing `nix.settings`.
this makes it easier to 'manually' serialize `nix.settings`, which is
relevant in e.g. safely handling sensitive values such as that of
`nix.settings.access-tokens`.
Signed-off-by: cinereal <cinereal@riseup.net>
`python3.pkgs` does not splice, so with a package-set setup to
cross-compile, we end up building xmltodict for the host platform. By
using `python3Packages.xmltodict` instead, we only use xmltodict for the
build platform, which is all that is needed for generating XML documents
at build-time.
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](8616af08d9/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
After final improvements to the official formatter implementation,
this commit now performs the first treewide reformat of Nix files using it.
This is part of the implementation of RFC 166.
Only "inactive" files are reformatted, meaning only files that
aren't being touched by any PR with activity in the past 2 months.
This is to avoid conflicts for PRs that might soon be merged.
Later we can do a full treewide reformat to get the rest,
which should not cause as many conflicts.
A CI check has already been running for some time to ensure that new and
already-formatted files are formatted, so the files being reformatted here
should also stay formatted.
This commit was automatically created and can be verified using
nix-build a08b3a4d19.tar.gz \
--argstr baseRev b32a094368
result/bin/apply-formatting $NIXPKGS_PATH
To keep assumptions about pkgs.formats.yaml in tact we pin it to a format
that will emit YAML 1.1.
Future usage should specify the YAML version explicitly.
Previously, setting listsAsDuplicateKeys or listToValue would make it so
merging these treat all values as lists, by coercing non-lists via
lib.singleton. Some programs (such as gamemode; see #345121), allow some
values to be repeated but not others, which can lead to unexpected
behavior when non-list values are merged like this rather than throwing
an error.
This now makes that behavior opt-in via the mergeAsList option. Setting
mergeAsList (to either true or false) without setting either
listsAsDuplicateKeys or listToValue is an error, since lists are
meaningless in this case.
