2026-10-04 NixBench 3.0 calibration

This study collects the first calibration evidence for corpus 3.0.0. Five configurations run the full 30-task corpus three times each in fresh-home bubblewrap sandboxes. Every 3.0 task is calibrating, so these results are calibration evidence, not leaderboard claims. They feed calibration-report and the activation review described in Benchmark governance.

Study design

• Trials: 3 full-corpus trials per configuration, 15 in total. Three

configurations with three valid observations per task is the minimum for activation review.

• Corpus: nixbench-public 3.0.0, 30 tasks, digest

0d4e05063591d17d598b9620543b57f03224db96271d403f03a7b56c25da9049 (python3 bench.py corpus-id --json). Git revision a20499aba85381fd518ab7a8c3688ed990bd8366.

• Isolation: linux-bwrap-fresh-home-v1, through the

codex-json-fresh-home-bwrap and claude-json-bwrap adapters. Each task gets a fresh home with login material only: no user skills, AGENTS.md or CLAUDE.md, settings, plugins, MCP servers, or session history. Network is enabled for the model connection. There is no Nix daemon socket, and the vendored nixpkgs lib is on NIX_PATH as <nixpkgs/lib>.

• Tools: web search and web fetch are disabled for both agents. Codex runs

with --disable apps. Claude Code runs with --setting-sources user and --strict-mcp-config, and with no --effort flag, so each model uses its default effort.

• Agent timeout: 300 seconds per task for every configuration. Evaluator

timeouts come from task metadata (60 seconds).

• Protocols: protocols/fresh-home-2026-10/bwrap/<key>.toml. Wrapper prompt:

protocols/agent-wrapper.txt.

• Runner: scripts/run_study_matrix.py

with --trials 3 --isolation bwrap. One worker per configuration runs concurrently; trials run sequentially inside each worker. The matrix started at 2026-10-03T22:48:05Z.

The exact agent commands, flags, and the namespace policy are documented in Running Agents. A trial that stops part-way is recorded as an excluded attempt and rerun in full. Task cells from different runs are never stitched into one trial.

Model IDs for Codex configurations are recorded with model_identity_evidence = "vendor-api-direct". Claude Code configurations use router-alias: the recorded ID names the requested alias, not an independent attestation of the upstream model.

Analysis plan

When all five configurations have three valid trials:

python3 bench.py study-matrix --studies-dir results/studies \
  --corpus-digest 0d4e05063591d17d598b9620543b57f03224db96271d403f03a7b56c25da9049 \
  --json
python3 bench.py calibration-report --studies-dir results/studies

Report per configuration the task pass rate with its Wilson interval, pass@k and pass^k, timeouts, and invalid attempts. Report per task the pooled solve rate, empirical difficulty, and failure set. List every task labelled saturated (pooled solve rate of 0.9 or more) for the activation review. Discrimination needs 20 valid observations across two configurations; with 15 observations per task it is expected to be unavailable.

Results

All five configurations completed three valid trials of thirty tasks: 450 valid task-trial observations, zero timeouts, zero invalid or incomplete attempts. Corpus digest 0d4e05063591d17d598b9620543b57f03224db96271d403f03a7b56c25da9049. Study summaries are archived under evidence/2026-10-04-nixbench-3-calibration/ and the site reads site/src/data/study-matrix.json.

Configurations

The top four intervals overlap; they are not ranked against each other. GPT-6 Luna is separated from all four. Cells of the same task are not independent, so the intervals are descriptive.

Tasks

Pooled solve rate over all 15 observations per task. Discrimination is the leave-one-task-out point-biserial computed with the exploratory threshold of 15 observations (the default is 20, so the committed site data shows it as unavailable).

Findings

• **The corpus is still too easy for frontier models.** Twenty-four of thirty

tasks are saturated (pooled solve rate at or above 0.9). Only six tasks carry any discrimination, and five of those discriminate only because GPT-6 Luna fails them. The top four configurations differ by at most three observations out of ninety.

• **Evaluator dispute: editor-project-state-isolation.** Its discrimination

is negative: the strongest configurations fail it while Luna passes. The evaluator's fake coreutils set (tests/probe.py, BASIC_TOOLS) did not include pwd, so any wrapper calling ${coreutils}/bin/pwd failed every launch. Re-scoring the 15 recorded trials against a repaired probe shows six were wrongly failed: five all-false trials (Opus 5.5 once, GPT-6.1 Sol twice, GPT-6 Astra twice) and one partial failure where Sol piped pwd -P into sha256sum without pipefail. All six pass the repaired evaluator. The task was quarantined for this release; the repair (complete coreutils set plus two new passing fixtures) landed afterwards and the task returns to calibrating for 3.1 under its new digest. package-offline-test-selection also has slightly negative discrimination and one Sonnet failure that should be reviewed before activation.

• **Sandbox and attestation held.** Every observation is valid and attested.

Each fresh home contained only login material. Codex authenticated directly with its auth.json; Claude Code went through the local router.

• **Cost axis.** Claude Code configurations used 16–25 agent seconds per task;

Codex configurations used 47–81 seconds at the same pass rates.

Decisions

• The release stays calibrating. No task is activated from this study.
• editor-project-state-isolation moves to quarantined.
• All other calibration records are recorded with decision pending.

Next steps

1. Retire or rework the 24 saturated tasks; keep them in the archive for

regression use.

1. Author the next round from the reserve pool in

task-candidates-designed.md and task-candidates-community.md, prioritising tasks that the strongest models failed here: laziness and type error boundaries, offline test selection, runtime tool wrapping, and fixed-output identity.

1. Repair the quarantined evaluator and recalibrate.
2. Rerun with five trials per configuration once the corpus has at least ten

tasks in the mixed band.