Reproducibility

NixBench is built to make benchmark runs inspectable, but full reproducibility still depends on the agent, model, machine, and task corpus version.

What The Harness Records

For each task, the harness records:

• Agent command.
• Agent return code.
• Agent timeout status.
• Agent duration.
• Evaluator command.
• Evaluator return code.
• Evaluator timeout status.
• Evaluator duration.
• Diff from starter to final workdir.
• Structured result JSON.

The aggregate run writes results/<run-id>/summary.json.

Recommended Run Metadata

When publishing or comparing results, record:

• NixBench git commit.
• Agent command.
• Agent version.
• Model name.
• Timeout.
• Host OS and architecture.
• Whether network access was available.
• Whether Nix was configured with flakes enabled.

Determinism Boundaries

The bundled tasks are mostly deterministic because evaluators use local Nix evaluation and fake builders. Agent behavior is not deterministic unless the agent and model expose a reliable deterministic mode.

For serious comparisons, run each model multiple times and report:

• pass rate
• mean score
• median score
• timeout rate
• common failure classes

Versioning The Corpus

Changing a task prompt, starter, evaluator, or reference solution changes the benchmark. Treat corpus changes as benchmark-version changes.

Suggested policy:

• Patch version: docs or harness-only changes.
• Minor version: new tasks added, no existing tasks changed.
• Major version: existing task behavior or scoring changed.

Avoiding Leakage

The hidden evaluator is intentionally outside the copied workdir. Do not give the agent paths to tasks/<id>/tests/check.sh. Do not include hidden assertions in prompt.md.

For public leaderboards, use a private hidden corpus in addition to the public corpus.