Add two shouldPass tests that catch known yj regressions:
- tomlHeterogeneousArray: arrays mixing scalars and attrsets crash yj
with 'unexpected primitive type' (NixOS/nixpkgs#511970)
- tomlCommaInKey: keys containing commas are silently truncated
because yj stores TOML keys in Go struct tags where commas are
option separators (sclevine/yj#52)
Expected values are taken from go-toml's jsontoml output.
Both tests fail under the current yj backend.
`remarshal` is a Python application; evaluating its derivation forces a
large slice of `python3Packages` (rich, ruamel-yaml, cbor2, tomlkit and
their transitive build/check inputs — ~150 distinct python-modules,
~2700 derivations). 95 NixOS modules use `formats.toml`, so any system
enabling one of those services pays this on every `nixos-rebuild`.
`yj` is a small Go binary that does the same JSON→TOML conversion. For
all values accepted by `(formats.toml{}).type` the output is
semantically identical (verified via `tomllib`).
Reproducer (best of 3, `NIX_SHOW_STATS=1 nix-instantiate -E
'(pkgs.formats.toml{}).generate "x" {a.b=1;}'`):
cpuTime nrFunctionCalls nrThunks drv-closure
before (remarshal) 0.69s 634,117 1,184,499 2722 paths
after (yj) 0.36s 271,308 566,981 1481 paths
-49% -57% -52% -46%
Runtime closure of the converter: remarshal 202.3 MiB → yj 5.2 MiB.
Differences in output:
- yj emits intermediate table headers for deeply-nested keys (`[a]`
`[a.b]` `[a.b.c]` rather than just `[a.b.c]`). Valid TOML, parses
identically.
- yj renders very small/large floats as decimals rather than scientific
notation (`1e-300` → 300-digit literal). Valid TOML, parses
identically.
- Direct callers of `generate` that pass `null` (which the format type
already rejects) get key-dropped instead of a build-time error.
NixOS modules are unaffected since the option type filters this at
eval time; several modules already `filterAttrs (v: v != null)` for
exactly this reason.
Tested:
- `nix-build pkgs/pkgs-lib/tests -A formats` passes
- arrays of tables, nested tables, string escaping, int64, unicode
keys/values, mixed scalar+table siblings round-trip through `tomllib`
identically to remarshal output
adds `nixConf` (to generate `nix.conf` files) as a file format to
`lib.formats`, and uses it for serializing `nix.settings`.
this makes it easier to 'manually' serialize `nix.settings`, which is
relevant in e.g. safely handling sensitive values such as that of
`nix.settings.access-tokens`.
Signed-off-by: cinereal <cinereal@riseup.net>
Format all Nix files using the officially approved formatter,
making the CI check introduced in the previous commit succeed:
nix-build ci -A fmt.check
This is the next step of the of the [implementation](https://github.com/NixOS/nixfmt/issues/153)
of the accepted [RFC 166](https://github.com/NixOS/rfcs/pull/166).
This commit will lead to merge conflicts for a number of PRs,
up to an estimated ~1100 (~33%) among the PRs with activity in the past 2
months, but that should be lower than what it would be without the previous
[partial treewide format](https://github.com/NixOS/nixpkgs/pull/322537).
Merge conflicts caused by this commit can now automatically be resolved while rebasing using the
[auto-rebase script](8616af08d9/maintainers/scripts/auto-rebase).
If you run into any problems regarding any of this, please reach out to the
[formatting team](https://nixos.org/community/teams/formatting/) by
pinging @NixOS/nix-formatting.
After final improvements to the official formatter implementation,
this commit now performs the first treewide reformat of Nix files using it.
This is part of the implementation of RFC 166.
Only "inactive" files are reformatted, meaning only files that
aren't being touched by any PR with activity in the past 2 months.
This is to avoid conflicts for PRs that might soon be merged.
Later we can do a full treewide reformat to get the rest,
which should not cause as many conflicts.
A CI check has already been running for some time to ensure that new and
already-formatted files are formatted, so the files being reformatted here
should also stay formatted.
This commit was automatically created and can be verified using
nix-build a08b3a4d19.tar.gz \
--argstr baseRev b32a094368
result/bin/apply-formatting $NIXPKGS_PATH
Previously, setting listsAsDuplicateKeys or listToValue would make it so
merging these treat all values as lists, by coercing non-lists via
lib.singleton. Some programs (such as gamemode; see #345121), allow some
values to be repeated but not others, which can lead to unexpected
behavior when non-list values are merged like this rather than throwing
an error.
This now makes that behavior opt-in via the mergeAsList option. Setting
mergeAsList (to either true or false) without setting either
listsAsDuplicateKeys or listToValue is an error, since lists are
meaningless in this case.
The new format is based on the existing wrapper and generates an INI file with an unnamed global section at the top as is used by *stunnel* for instance.
Technically the INI format is a subset of this however testing, type checking, and API guarantees profit from two separate generators.
Co-authored-by: tim-tx <tim-tx@users.noreply.github.com>
Signed-off-by: benaryorg <binary@benary.org>
Tests using `shouldFail` (new) enable testing for whether the type system catches any unintended uses (missing parameters or unavailable data types in the format).
It should not be necessary to test for all possible outliers for each format, however format-specific tests (for instance those using a rigid submodule structure) can ensure that common mistakes err out instead of being silently discarded, while also allowing test-driven development of sorts.
Signed-off-by: benaryorg <binary@benary.org>
The way `(lib.formats.yaml {}).generate` generates YAML is compliant
because on YAML 1.2 spec JSON is a subset of YAML but it bugs people's
minds and can lead to problems with software that is not compatible with
YAML 1.2.
This commit also changes the test of the generation function. Data
validation/typing remains the same.
See #133802.
Signed-off-by: lucasew <lucas59356@gmail.com>
