Final corpus digest: f588fc8d9dcbaadeb8f342d733374ec334433fed05ec44a8c143074d3d6ed5a1.
This release changes the accepted-solution and scoring boundaries. Do not pool its trials with prerelease or historical binary-scored results.
For provenance, the pre-repair 2.0.0-dev corpus digest was d7364a62046c74e96ab4cf19db48ecd6ec14f8b83b22814fdae7ba17c6c5118d. The evaluator-repaired prerelease before rubric migration had digest 126f39197587ffd2ce22580945f8fed6c13c5bf094d6e0afb493de80222d8495. The Plan 004 review prerelease had digest 0a0c79dd3407e85077a7522f63b6b3101b7e254457e83908bc388642407bc7bf.
Prompt and evaluator changes
debug-network-false-leaddocuments its observation schema and
machine-graded fact codes while leaving human evidence wording free-form.
issue-report-qualityrecords expected and actual outcomes in explicit
status fields while leaving explanatory prose free-form.
container-native-vs-ocidocuments the neutrallibandpkgsinterface
available to nested container module functions.
- Evaluators accept documented equivalent module, package, dependency,
quoting, and report representations. Targeted invalid fixtures still reject behavior outside each task contract.
Rubric scoring
All 29 tasks declare objective criteria totaling 100 points. Evaluators emit schema-2 criterion booleans, including for ordinary candidate parse and evaluation failures. The harness computes points and failure classes from task metadata. Contract cases name the criterion they exercise, and the contract suite covers every required criterion.
Results separate measurement validity from task outcome. Invalid and incomplete study attempts remain in the attempts ledger but do not enter trials or estimates. Historical scalar scores remain readable as legacy-binary. Publication requires an explicit compatibility flag and does not convert or pool those scores with criterion-scored runs.
Criterion contract hardening
The unreleased 2.0.0 worktree now uses evaluator contract schema 3. Every fixture declares its complete expected criterion vector, every required criterion has a targeted rejecting fixture, and duplicate rejecting candidate digests cannot count as independent evidence without an explicit rationale. Coupled failures are recorded rather than hidden.
Evaluator totality repairs prevent ordinary missing nested fields from erasing unrelated partial credit. Contract tests, corpus-health, and release-check use the same production loader and executor, apply delete and rename operations identically, repeat complete vectors for determinism, and reject mislabeled criteria, evaluator error logs, active known-issue skips, or missing alternative passing fixtures.
This remained version 2.0.0 because the repository's release plan identifies 2.0.0 as an unpublished worktree release and forbids public 2.0.0 trials before these hardening plans finish. If a 2.0.0 artifact had already been published, this evaluator and scoring-boundary migration would instead require 3.0.0. Historical studies are not rescored or pooled across corpus digests.
Completion attestation
Complete protocols require a registered completion adapter whose executable digest is part of the configuration identity. The Codex adapter derives preflight, completion, transport, and launcher-exit state from native JSONL events. The harness snapshots and removes its status before evaluation. The adapter removes harness-private paths from the model-controlled child environment. Same-UID trust remains provisional. Held-out publication instead requires the approved bubblewrap adapter and manifest-bound isolation evidence.
Plan 004 review fixes
- Debug-network scoring now checks the published false-lead discipline record,
and Rust source and metadata preservation are explicit public requirements.
- Flake whole-evaluation scoring no longer repeats package, app, check, or
development-shell assertions owned by other criteria.
- Per-system output failures now lose only their owning criterion; successful
flake import, output evaluation, and the empty-input assertion earn the independent whole-evaluation credit.
- Flake input package selection observes Linux and Darwin package composition
independently. Selecting .default instead of the named input output loses only the named-output criterion.
- Zero-trial attempt ledgers do not block study resumption or valid sibling
exports. Harness exceptions and interruptions checkpoint excluded attempts.
- Evaluator exit status follows required criteria, so failed optional criteria
retain partial credit without creating an invalid measurement.
Calibration migration
The final post-contract corpus has no checked current-digest empirical model observations. No paid or external runs were authorized for this migration, so all 29 release-controlled tasks are honestly marked calibrating. The schema-3 release manifest retains their release identities but has an empty active task set and is not eligible for publication or active leaderboard claims.
Activation now requires strict schema-3 evidence from at least three materially different configurations, with at least one valid observation per task in each configuration, plus manual review of accepted alternatives, evaluator disputes, difficulty evidence, timeouts, invalid attempts, and unavailable statistics. Authorized operators must collect and review that evidence later; historical or older-digest studies do not satisfy the gate.
Verification
The final Plan 004 gate passed all 141 tests. Reference validation passed all 29 tasks at full score, starter validation rejected all 29 tasks with valid criterion results, and git diff --check passed.
Release governance
The checked release manifest records the corpus digest, composition, scoring and reporting versions, required protocol schema, and canonical release-gate digest. Release checks now recompute reference, starter, contract, determinism, validity, and runtime evidence. Publication checks separately validate study identity and trial completeness.
The public corpus remains the inspectable development corpus. Private held-out publication requires the approved linux-bwrap-v1 launcher and redacted aggregation rules. This repository contains only the public tooling and an empty private-corpus template command. It contains no active private tasks.
The launcher uses a fixed in-namespace PATH and generic preflight that does not expose forbidden host paths through process arguments or environment. The cleared inner command is PID 1, which prevents the bubblewrap reaper from exposing the outer environment through /proc/1/environ. The runner rejects workspace symlinks that escape the sealed workspace before evaluation. Publication derives visibility, adapter trust, task count, and isolation requirements from the checked release manifest. Cached health reports bind their canonical evidence digest, and the release manifest records the active task set and task digests used to verify release change claims.
Trusted adapter identity now includes each adapter's declared local bundle. For codex-json-bwrap, that bundle is the launcher script, nixbench/isolation.py, and the reviewed launcher TOML. The legacy entry-point digest keeps its original meaning; a separate bundle digest is included in configuration identity and in schema-3 release manifests. Private publication compares the study, live adapter registration, and release manifest, while older manifests remain explicitly incompatible. This identity covers harness-side isolation behavior, not provider code, model weights, or the external model executable.
Held-out runs use the neutral host staging path /tmp/nixbench-isolated-<random>/work. The preflight verifies this bind source and the fresh home mount through /proc/self/mountinfo, so mount metadata does not disclose private task or experiment identities. External agent executables use a separate neutral /tmp/nixbench-isolated-agent-<random>/agent bind source instead of their original host path.
Public site export refuses active held-out and retired studies. Redacted publication export first runs the publication gate, derives suppression thresholds from unique valid task cells, and emits only allowlisted aggregate scalars. Active quarantined tasks fail the release gate and cannot contribute to headline scores.
Calibration review follow-up
Initial three-configuration calibration exposed evaluator exceptions and unstated shape assumptions in four tasks. Before activation, the package task was expanded to accept final-attribute derivations and any non-empty platform list; the issue-report, mutable-state, and service-option prompts now state the shapes their rubrics require; and those evaluators preserve unrelated partial credit when one field is missing or has the wrong type. The observed model candidates were added as permanent contract fixtures. Calibration registry validation now accepts only canonical cfg-<sha256> identities and reports empirical difficulty rather than the inverse solve-rate label. A second review round added top-level module-attrset support for native containers and made the development-shell contract explicit about newline-safe preservation of an existing NIX_CONFIG; both observed candidates are retained as regression fixtures.
The final post-fix corpus was calibrated with one complete observation from each of three declared configurations: GPT-5.6 Sol/high, Terra/medium, and Luna/low through direct Codex CLI execution with Pi extensions disabled. Their whole-corpus scores were 2875/2900, 2875/2900, and 2850/2900 respectively. All observations were valid and complete, with no timeouts or infrastructure events. The repeated Rust failure was a correctly isolated malformed fixed hash; the Luna development-shell failure was a correctly isolated malformed NIX_CONFIG hook. Every task received explicit review and approval. Because there is only one observation per configuration, model identity evidence is unverified, and same-UID execution trust is provisional, these calibration results are descriptive activation evidence—not a stable comparative ranking.
Calibration evidence archive
The exact schema-3 study summaries, run summaries, task scores, diffs, evaluator logs, reviewed protocols, and checksum manifest are published with the GitHub 2.0.0 release as nixbench-2.0.0-calibration-evidence.tar.zst. Raw agent transcripts are excluded because they contain machine-local paths and unrelated user-level agent configuration; their exclusion does not change the study-summary hashes recorded by the calibration registry.