Release date: 2026-10-04. Major release. State at release: calibrating.
Why
The 3.1.0 calibration (2026-10-04-nixbench-3.1-calibration.md) widened the spread between configurations but left nine tasks saturated and the top four configurations tied. Frontier agents solved every task whose defect surfaced as an evaluation error, because they iterate on nix-instantiate until it is clean. The one task they all failed had a requirement invisible to evaluation. 3.2 is built entirely from that pattern.
What changed
- **Silent failures.** Six new eval-track tasks whose starters evaluate
cleanly against the pinned nixpkgs and whose defect appears only at runtime. The requirement is documented in the pinned tree (release notes, option descriptions, module source), and the prompt describes the operator's observation, never the mechanism. Each is coupled with a multi-host regression trap so that the natural fix on one host must not change a sibling. Source research: silent-failure-changes-2026-10.md.
- **VM-graded track.** Three tasks are graded by running NixOS VM tests
against the candidate repository with testers.runNixOSTest on the pinned tree. The evaluator builds the baseline and candidate tests in parallel, maps runtime assertions to criteria through a verdict file so one failure does not zero the rest, and exits 2 for infrastructure problems. They declare timeout_seconds = 600; observed cold runtimes are 45 to 155 seconds and cached runs under 10 seconds. The agent cannot run these tests inside the sandbox; the prompt quotes the deploy pipeline's output.
- **Corpus.** Nine new tasks. The nine saturated 3.1 tasks moved to
archive/3.1/ with deprecation records. Twelve 3.1 tasks remain.
Operational note
VM-track evaluators run on the host with the Nix daemon and may fetch from the binary cache the first time a closure is needed; later runs hit the store. A host without /dev/kvm records invalid measurements for those tasks.
Lifecycle
Every task is calibrating. Activation requires the calibration minimums in benchmark-governance.md and a saturation review.
Compatibility
3.1.0, 3.0.0, and 2.0.0 trials stay bound to their digests and are never pooled with 3.2.0 results.