The 2.0 evaluators used a fake lib and fake module plumbing. Many checked the syntactic shape of a candidate instead of what it evaluates to, so they rejected valid Nix. The 3.0 evaluators make three changes:
- **Real nixpkgs
lib.** Evaluators import a pinned, vendored copy
(vendor/nixpkgs-lib, see PIN.md) through $NIXBENCH_VENDOR_LIB, which the runner exports to evaluators only.
- **Real module system.** Module tasks run
lib.evalModules. A minimal option
skeleton declares only what the evaluator inspects, with a lazy freeform root. Assertions read the resolved config, so mkIf at any level, mkMerge, mkDefault, config = versus shorthand, and module functions with or without ... all behave as they do in NixOS.
- **Execution instead of regexes.** Shell snippets such as
ExecStart,
installPhase, and fixup hooks run in a sandbox (nixbench_sandbox.py: scrubbed environment, a timeout, and a bubblewrap read-only host view with no network when bubblewrap is available). The evaluator then checks the effects.
Cross-cutting problem: all-false fallbacks
builtins.tryEval does not catch missing-attribute or type errors. In 2.0, one bad field aborted the single nix eval, and the fallback then wrote all-false criteria, zeroing unrelated criteria. Every 3.0 evaluator uses nixbench_score (vendor/nixpkgs-lib/nixbench-evaluator.sh), which evaluates each criterion in its own Nix process. An aborted criterion is false with a note, and the other criteria are still measured. Nix diagnostics go to a private log, not the evaluator log, so the release gate's error-log check stays meaningful. Set NIXBENCH_EVALUATOR_DEBUG=1 to print them.
Per-task findings
Every changed evaluator has at least one new passing fixture that 2.0 rejected. Fixture names are under contracts/<task>/.
| Task | Problem found | Fix | Valid alternatives now accepted (fixture) |
|---|---|---|---|
| container-native-vs-oci | Host module called with {}, so { lib, ... } modules failed. Nested config called with a fake lib. Bind mount had to be keyed "/dev/bus/usb" with an explicit hostPath. | Real evalModules for the host and for the nested deferredModule config. Uses NixOS container defaults (isReadOnly defaults to true, hostPath defaults to mountPoint). | Host-level lib.mkMerge/mkDefault; a named bind mount with mountPoint; a nested config of any module shape (named-bind-mount-with-module-helpers). New reject: omitting isReadOnly relies on the read-only default (usb-bind-mount-relies-on-read-only-default). |
| debug-infinite-recursion | A grep ban on lib.optionals zeroed evaluates-with-lib. The explicit lib was fake. | Calls the file with a minimal { optional } lib and with the real nixpkgs lib; both must evaluate to the same value. The ban is removed. | Feature detection such as lib.optionals or (...) (feature-detects-lib-optionals). |
| debug-network-false-lead | Whole-file evaluation with an all-false fallback. | Per-criterion processes. | No shape changes needed. |
| devshell-tooling-contract | Tools only counted from packages. NIXBENCH_FORMATTER only as a top-level attr. NIX_CONFIG only via a shellHook export. | Tools may be in packages, nativeBuildInputs, or buildInputs. Variables may be top-level or under env. NIX_CONFIG may be an attribute. | nativeBuildInputs + env.NIXBENCH_FORMATTER + env.NIX_CONFIG (native-build-inputs-and-env-attributes). |
| fetcher-source-pin | Exact leaveDotGit == false required the attribute even though false is the fetcher default. | (leaveDotGit or false) == false. | Omitting leaveDotGit (omits-default-leave-dot-git). |
| fhs-binary-wrapper | Only buildFHSUserEnv was passed, so the current nixpkgs name buildFHSEnv failed with a missing argument. runScript had to be the literal "vendor-tool". Module surfaces got a fake lib. | callPackage-style arguments that provide both names, plus the real lib. runScript may also be the AppImage's bin/vendor-tool store path. | buildFHSEnv with runScript = "${appimage}/bin/vendor-tool" (build-fhs-env-runs-appimage-binary). |
| flake-input-package-selection | The fake pkgs lacked stdenv.hostPlatform.system, the current replacement for the deprecated pkgs.system. | Added to both probe package sets. | inputs.x.packages.${pkgs.stdenv.hostPlatform.system} (host-platform-system). |
| flake-per-system-outputs | Whole-file evaluation with an all-false fallback. | Per-criterion processes. | No shape changes needed. The lib.systems order stays because it is a declared output value. |
| home-manager-extra-special-args | The Home Manager settings had to be an inline attrset in modules with a literal home-manager key, so module functions, config =, and mkMerge failed. The system argument was required, so nixpkgs.hostPlatform failed. The HM module was a marker value. | Fake nixosSystem runs real evalModules. The HM NixOS module is a real module that declares its options. The user module is evaluated as a deferredModule. | nixpkgs.hostPlatform plus an HM module function using lib.mkDefault and a shared specialArgs binding (host-platform-and-module-function). |
| home-manager-wsl-module-import | Raw attribute reads (cfg.wsl.enable), so mkMerge, module functions, and user module functions failed. | Real evalModules. The supplied HM module declares its options and marks itself imported. An ambient <home-manager/nixos> resolves to an empty module so the other criteria are still measured. | Top-level lib.mkMerge and a user module function (mkmerge-and-user-module-function). |
| home-manager-xdg-files | Result had to be a raw attrset, so module functions and lib helpers failed. | Real evalModules with Home Manager file option types (text, source, executable, onChange). Imperative-command detection in hooks stays text-based: running hooks safely would need a copy-on-write overlay of the whole host. | A module function using lib.mkMerge/mkDefault (returns-module-function-with-lib-helpers). New reject: home.activation (activation-script-creates-directory). |
| issue-report-quality | Whole-file evaluation with an all-false fallback. | Per-criterion processes. | No shape changes needed. |
| lang-attrsets-normalize | Whole-file evaluation with an all-false fallback. | Per-criterion processes. | No shape changes needed. List order follows builtins.attrNames, which is Nix semantics. |
| module-path-composition | Whole-file evaluation with an all-false fallback. | Per-criterion processes. | No shape changes needed. The import order is a stated requirement. |
| module-service-options | The whole config had to be one lib.mkIf wrapper. Fake option types were compared by string. allowedTCPPorts == [port] exact equality. Grep ban on lib.concatStringsSep. Regex match on ExecStart. | Real evalModules. Probes are enabled (two port/package/argument sets), disabled, and unset. Option types are checked by behaviour (port accepts 65535 and rejects 65536 and strings). Another module pre-opens port 22, which must survive. ExecStart is split shell-style and its argv compared. | Per-attribute mkIf inside mkMerge, types.ints.u16, lib.escapeShellArgs, lib.getExe (mkmerge-with-per-attribute-mkif); lib.concatStringsSep (service-uses-lib-concat-strings-sep, formerly a reject). New reject: mkForce on the firewall list (firewall-force-replaces-other-ports). service-declares-wrong-port-type now uses types.int, because with real types types.str also breaks the integer probes. |
| module-stale-option-migration | Module called with {}, so { lib, ... } modules failed. Raw attribute reads. | Real evalModules. Stale option trees are declared freeform so leftover definitions stay visible. | lib.mkDefault and nested option attrsets (module-helpers-and-nested-graphics). New reject: a leftover hardware.opengl alias (keeps-stale-opengl-alias). |
| module-system-boundaries | Modules called with lib = {}. Package lists compared as exact multisets. | Each module is evaluated by evalModules against its own system's skeleton, so leaks into another system's options land in the freeform root. Package lists are checked by containment. | Shared module constructor with mkMerge/mkIf/mkDefault (modules-use-lib-helpers). |
| mutable-config-home-manager | Result had to be a raw attrset. mutableState == [ profileDir ] exact. | Real evalModules. mutableState is checked by containment. Policy JSON is parsed. | A module function using lib.generators.toJSON and extra policies (module-function-with-lib-generators). |
| nushell-command-not-found | Fake mkIf/mkMerge. The disabled probe required the whole config to equal {}. | Real evalModules with types.lines options. Gating is probed with each enable-flag combination. The Nushell hook body is still checked as text because there is no offline Nushell interpreter. | A combined mkIf condition inside mkMerge, with lib.getExe' (mkmerge-with-combined-condition-and-getexe). |
| overlay-module-boundary | Module called with only pkgs (no lib). Hash compared as the exact base16 string. buildInputs exact list. The fake mkDerivation lacked finalAttrs. | Real evalModules for the module. Hashes compared with builtins.convertHash. buildInputs must contain the final inputs and none of the prev ones. | SRI hash, reordered inputs, lib.mkMerge/lib.getExe module (sri-hash-and-lib-module-helpers). |
| overlay-override-package | The fake overrideAttrs failed on finalAttrs: previousAttrs: overrides. meta.broken == false required. | overrideAttrs built on lib.extends/lib.toExtension, as in mkDerivation. meta.broken may be absent. | Two-argument overrides and removing broken (final-attrs-override-and-removed-broken-flag). |
| package-name-lookup-contract | Whole-file evaluation with an all-false fallback. | Per-criterion processes. | No shape changes needed. The exact package set is the task's point. |
| package-python-application | Required nativeBuildInputs, pytest, and pytestFlagsArray (exact lists). Fake licenses. No finalAttrs. | Accepts build-system/dependencies/pytestCheckHook/enabledTestPaths/pytestFlags/finalAttrs and checks by containment. Real lib.licenses. | Current nixpkgs Python conventions (current-nixpkgs-python-conventions). |
| package-stdenv-cli | Regex on installPhase. Fake lib. Exact makeFlags list. rev only. | installPhase runs in a fake build tree (stub runHook, installBin, installShellCompletion, and make honouring makeFlags). It must install the built binary at $out/bin/tinygrep. Real lib. Containment checks. tag accepted. | installBin and tag (install-bin-helper-and-tag). New reject: installing to $out/libexec (install-targets-wrong-directory). |
| purity-wrapper-derivation | Grep ban on lib.getExe. A fake makeBinPath threw unless given exactly [ coreutils ]. .*HOME.* regex on installPhase (it also rejected HOME=$TMPDIR). | Real lib. installPhase runs with a recording makeWrapper and a sentinel HOME. The recorded wrapper must wrap ${bash}/bin/bash with coreutils/bin on PATH. Sentinel HOME or host /usr, /bin, /sbin paths reaching the wrapper fail. | lib.getExe bash (pure-wrapper-uses-lib-get-exe, formerly a reject); a literal ${coreutils}/bin PATH with extra flags (wrapper-literal-coreutils-bin-path). New rejects: wrapper-captures-build-home, wrapper-appends-host-path. |
| python-cuda-uv2nix-patch | Regex required addAutoPatchelfSearchPath in preFixup with the exact torch/lib path. hatchling only from nativeBuildInputs. CUDA_HOME only top-level. No finalAttrs. Bans also matched comments. | Build and fixup hooks run with a recording addAutoPatchelfSearchPath. Any path that recursively covers torch's lib directory in either probe passes. Accepts build-system and env.CUDA_HOME. Comment lines are excluded from the bans. | postInstall adding torch's site-packages recursively, build-system, dependencies, env.CUDA_HOME (recursive-search-path-and-current-conventions). |
| rust-no-network-build | Fake lib (licenses.mit = "MIT", identity getLib). cargoHash required. rev only. No finalAttrs. | Real lib. Accepts cargoLock, tag, and finalAttrs. | cargoLock, tag, buildFeatures, lib.makeLibraryPath (cargo-lock-tag-and-lib-helpers). |
| string-escaping-systemd | Regex on ExecStart that only accepted printf ... >> or tee -a. Raw module attrs. | Real evalModules. ExecStart is split like systemd (with $/${VAR} substitution) and run twice under real bash with random messages. Assertions: the message is expanded at runtime (seen in the log, stdout, or xtrace), and each run appends to the log. An evaluation-time sentinel environment catches builtins.getEnv capture. | echo in an indented string with ''${...} escapes and lib.escapeShellArg (echo-in-indented-string-with-escape-shell-arg). New reject: message-captured-at-evaluation-time. |
| xdg-portal-merge | Fake mkIf/mkAfter/mkForce marker objects. Required xdg.portal = lib.mkIf ... at exactly that level. Merging was simulated by the evaluator. | Real evalModules with another module that already defines portals, config packages, and common.default. mkForce and self-reading definitions fail through real merging and recursion. | Per-attribute mkIf, plain lists without mkAfter, per-desktop config.hyprland.default (per-attribute-mkif-with-plain-lists). New reject: reading config.xdg.portal.extraPortals while defining it (reads-portal-list-while-defining-it). |
Remaining source-text checks
These are explicit prompt prohibitions on impurity or entrypoints, not bans on helpers:
builtins.getEnvinstring-escaping-systemdand
purity-wrapper-derivation.
- Quoted
/usr/binand/binstrings inpurity-wrapper-derivation. <home-manager/nixos>andhomeManagerConfigurationin
home-manager-wsl-module-import.
All of them ignore comments.
Known limitations
ExecStartexecution does not expand systemd specifiers such as%s.
Real systemd would expand %s in printf "%s\n" to the user's shell, which affects the reference too.
- Nushell hook text and Home Manager hook commands are still checked as text
(see the table).
- Without a working bubblewrap, the sandbox falls back to a scrubbed
environment with a timeout and a private working directory only.
- The vendored lib and the evaluator support files sit outside
tasks/, so
they are not hashed into the corpus digest. Each evaluator binds to the pin with nixbench_init <commit>. tests/test_vendor_lib.py checks that every evaluator uses the recorded commit and that lib/ matches the recorded NAR hash.