NixBench

Benchmark results

Configuration means, individual trials, and honest uncertainty.

Compare integer tasks solved and normalized time per task. Historical and current corpora stay separate, and a confidence interval appears only when repeated trials support one.

Current-corpus coverage16 configurations72 trials

models
9
configurations
31
recorded trials
87
task rows
29

Compare the signal first. Inspect the scatter second.

Ordered effort paths lead the view. Select a model for uncertainty and effort labels, or reveal every trial to inspect the underlying variation.

Corpus
Evidence
Y-axis
5 models · 16 configurations · 72 trials16 means shown14/16 replicated

Configuration means, with uncertainty

Mean tasks passed against mean agent seconds per task. Paths connect ordered effort configurations; select a model to reveal its 95% Student's t intervals and effort labels. The task axis focuses on 6–29 to make the observed differences legible. The time axis uses logarithmic spacing because observed runtimes span more than one order of magnitude. Individual trials are hidden in this summary view.

Focused: 6–29 tasksLog time axis↑ more tasks← less time
Ordered effort path + meanSingle observation or legacy composite; no CISelect a model below to reveal effort labels and uncertainty.
95% CI / observed
GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutmediumn=524.0 / 29
22.5–25.5observed 22–2538.5s18m 36s / corpus0
GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutxhighn=524.0 / 29
23.1–24.9observed 23–2560.7s29m 20s / corpus0
GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutlown=523.6 / 29
22.2–25.0observed 22–2527.4s13m 14s / corpus0
GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeouthighn=523.6 / 29
22.9–24.3observed 23–2447.4s22m 53s / corpus0
GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutmediumn=523.2 / 29
22.2–24.2observed 22–2430.5s14m 45s / corpus0
GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutmaxn=523.2 / 29
20.1–26.3observed 19–2581.4s39m 19s / corpus3
GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutlown=523.0 / 29
22.1–23.9observed 22–2426.6s12m 51s / corpus1
GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeouthighn=522.8 / 29
20.8–24.8observed 20–2438.4s18m 34s / corpus0
GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutxhighn=522.8 / 29
21.8–23.8observed 22–2448.8s23m 35s / corpus0
GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeouthighn=522.6 / 29
21.2–24.0observed 21–2444.4s21m 29s / corpus0
GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutxhighn=522.4 / 29
21.7–23.1observed 22–2353.9s26m 2s / corpus0
GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutmaxn=522.4 / 29
20.7–24.1observed 21–2475.0s36m 16s / corpus4
GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutlown=522.0 / 29
21.1–22.9observed 21–2325.4s12m 16s / corpus0
GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeoutmediumn=521.8 / 29
20.2–23.4observed 20–2328.4s13m 44s / corpus0
Gemma 4 26B-A4B QAT Q4_0 via OpenCode + llama.cpp29-task corpus · 1 recorded trial · 1h timeoutdefaultsingle run10.0 / 29
unavailableobserved 10–101869.1s15h 3m / corpus3
Ternary Bonsai 27B Q2_0 via OpenCode + llama.cpp29-task corpus · 1 recorded trial · 1h timeoutdefaultsingle run7.0 / 29
unavailableobserved 7–72478.7s19h 58m / corpus4
  1. GPT-5.6 Sol via Codex CLI29-task corpus · 4m timeoutmedium
    Evidence
    n=5
    Mean tasks
    24.0/29
    95% CI
    22.5–25.5
    Seconds / task
    38.5s
    Observed 22–25 tasks · 0 timeouts
  2. GPT-5.6 Sol via Codex CLI29-task corpus · 4m timeoutxhigh
    Evidence
    n=5
    Mean tasks
    24.0/29
    95% CI
    23.1–24.9
    Seconds / task
    60.7s
    Observed 23–25 tasks · 0 timeouts
  3. GPT-5.6 Sol via Codex CLI29-task corpus · 4m timeoutlow
    Evidence
    n=5
    Mean tasks
    23.6/29
    95% CI
    22.2–25.0
    Seconds / task
    27.4s
    Observed 22–25 tasks · 0 timeouts
  4. GPT-5.6 Sol via Codex CLI29-task corpus · 4m timeouthigh
    Evidence
    n=5
    Mean tasks
    23.6/29
    95% CI
    22.9–24.3
    Seconds / task
    47.4s
    Observed 23–24 tasks · 0 timeouts
  5. GPT-5.6 Luna via Codex CLI29-task corpus · 4m timeoutmedium
    Evidence
    n=5
    Mean tasks
    23.2/29
    95% CI
    22.2–24.2
    Seconds / task
    30.5s
    Observed 22–24 tasks · 0 timeouts
  6. GPT-5.6 Sol via Codex CLI29-task corpus · 4m timeoutmax
    Evidence
    n=5
    Mean tasks
    23.2/29
    95% CI
    20.1–26.3
    Seconds / task
    81.4s
    Observed 19–25 tasks · 3 timeouts
  7. GPT-5.6 Luna via Codex CLI29-task corpus · 4m timeoutlow
    Evidence
    n=5
    Mean tasks
    23.0/29
    95% CI
    22.1–23.9
    Seconds / task
    26.6s
    Observed 22–24 tasks · 1 timeouts
  8. GPT-5.6 Terra via Codex CLI29-task corpus · 4m timeouthigh
    Evidence
    n=5
    Mean tasks
    22.8/29
    95% CI
    20.8–24.8
    Seconds / task
    38.4s
    Observed 20–24 tasks · 0 timeouts
  9. GPT-5.6 Terra via Codex CLI29-task corpus · 4m timeoutxhigh
    Evidence
    n=5
    Mean tasks
    22.8/29
    95% CI
    21.8–23.8
    Seconds / task
    48.8s
    Observed 22–24 tasks · 0 timeouts
  10. GPT-5.6 Luna via Codex CLI29-task corpus · 4m timeouthigh
    Evidence
    n=5
    Mean tasks
    22.6/29
    95% CI
    21.2–24.0
    Seconds / task
    44.4s
    Observed 21–24 tasks · 0 timeouts
  11. GPT-5.6 Luna via Codex CLI29-task corpus · 4m timeoutxhigh
    Evidence
    n=5
    Mean tasks
    22.4/29
    95% CI
    21.7–23.1
    Seconds / task
    53.9s
    Observed 22–23 tasks · 0 timeouts
  12. GPT-5.6 Luna via Codex CLI29-task corpus · 4m timeoutmax
    Evidence
    n=5
    Mean tasks
    22.4/29
    95% CI
    20.7–24.1
    Seconds / task
    75.0s
    Observed 21–24 tasks · 4 timeouts
  13. GPT-5.6 Terra via Codex CLI29-task corpus · 4m timeoutlow
    Evidence
    n=5
    Mean tasks
    22.0/29
    95% CI
    21.1–22.9
    Seconds / task
    25.4s
    Observed 21–23 tasks · 0 timeouts
  14. GPT-5.6 Terra via Codex CLI29-task corpus · 4m timeoutmedium
    Evidence
    n=5
    Mean tasks
    21.8/29
    95% CI
    20.2–23.4
    Seconds / task
    28.4s
    Observed 20–23 tasks · 0 timeouts
  15. Gemma 4 26B-A4B QAT Q4_0 via OpenCode + llama.cpp29-task corpus · 1h timeoutdefault
    Evidence
    single run
    Mean tasks
    10.0/29
    95% CI
    unavailable
    Seconds / task
    1869.1s
    Observed 10–10 tasks · 3 timeouts
  16. Ternary Bonsai 27B Q2_0 via OpenCode + llama.cpp29-task corpus · 1h timeoutdefault
    Evidence
    single run
    Mean tasks
    7.0/29
    95% CI
    unavailable
    Seconds / task
    2478.7s
    Observed 7–7 tasks · 4 timeouts

Corpora are intentionally separated and time is normalized per task. The focused y-axis is explicitly labelled; Full scale restores the zero baseline. Lines show configuration order from lower to higher effort; they do not imply continuous scaling or monotonic treatment. See the reproducibility method and local OpenCode run provenance. Raw run IDs are shown in trial tooltips.

Current trial environments: 2 agent versions · 2 hosts · 2 repository revisions · network unknown · timeout budgets 240s, 3600s.

Coverage by model family, without picking a lucky winner.

Ranges summarize configuration means. Trial and replication counts show how much evidence sits behind each family.

Methodology
ModelCorpusMean task rangeSeconds / task rangeEffort settingsEvidence
codexGPT-5.526-task corpus19.0–22.0 tasks44.8–94.7slow, medium, high, xhigh4 trials · 4 configurations0/4 replicated · 0 timeouts
codexGPT-5.426-task corpus20.0–22.0 tasks41.0–92.4slow, medium, high, xhigh4 trials · 4 configurations0/4 replicated · 0 timeouts
codexGPT-5.4 mini26-task corpus19.0–21.0 tasks32.0–91.8slow, medium, high, xhigh4 trials · 4 configurations0/4 replicated · 4 timeouts
claudeClaude Opus 4.826-task corpus19.0–21.0 tasks20.8–50.9slow, high, xhigh3 trials · 3 configurations0/3 replicated · 0 timeouts
codexGPT-5.6 Sol29-task corpus23.2–24.0 tasks27.4–81.4shigh, low, max, medium, xhigh25 trials · 5 configurations5/5 replicated · 3 timeouts
codexGPT-5.6 Terra29-task corpus21.8–22.8 tasks25.4–48.8shigh, low, medium, xhigh20 trials · 4 configurations4/4 replicated · 0 timeouts
codexGPT-5.6 Luna29-task corpus22.4–23.2 tasks26.6–75.0shigh, low, max, medium, xhigh25 trials · 5 configurations5/5 replicated · 5 timeouts
opencodeGemma 4 26B-A4B QAT Q4_029-task corpus10.0–10.0 tasks1869.1–1869.1sdefault1 trials · 1 configurations0/1 replicated · 3 timeouts
opencodeTernary Bonsai 27B Q2_029-task corpus7.0–7.0 tasks2478.7–2478.7sdefault1 trials · 1 configurations0/1 replicated · 4 timeouts

Every task, shown against fixed baseline runs.

These are named comparison runs—not each model’s best row. Codex columns use xhigh effort; the historical Claude column preserves the original default composite.

Showing 7 of 7 model columns

Comparison contextHistorical 26-task and current 29-task baselines · 240-second per-task timeoutRows marked (+2) combine a 24-task run with two supplemental task artifacts.Inspect run provenance

5.5

GPT-5.5 · 26-task corpus · xhigh

22/264 failed

Average task time: 1m 35s

5.4

GPT-5.4 · 26-task corpus · xhigh

21/265 failed

Average task time: 1m 32s

mini

GPT-5.4 mini · 26-task corpus · xhigh

19/267 failed

Average task time: 1m 32s

opus

Claude Opus 4.8 · 26-task corpus · default

21/265 failed

Average task time: 59s

sol

GPT-5.6 Sol · 29-task corpus · xhigh

21/298 failed

Average task time: 55s

terra

GPT-5.6 Terra · 29-task corpus · xhigh

19/2910 failed

Average task time: 42s

luna

GPT-5.6 Luna · 29-task corpus · xhigh

19/2910 failed

Average task time: 49s

Pass/fail status and elapsed task seconds for the selected model columns.
TaskAreaGPT-5.526-task corpus · xhigh20260624T182835Z-4ad8b555 (+2)GPT-5.426-task corpus · xhigh20260624T190640Z-fa04a19c (+2)GPT-5.4 mini26-task corpus · xhigh20260624T194359Z-268b0abe (+2)Claude Opus 4.826-task corpus · default20260624T202141Z-881ef1e9 (+2)GPT-5.6 Sol29-task corpus · xhigh20260709T175817Z-cb86c575GPT-5.6 Terra29-task corpus · xhigh20260709T175820Z-36a5f2f2GPT-5.6 Luna29-task corpus · xhigh20260709T175826Z-b8e5f041
container-native-vs-ociModulesFail53.0sFail48.3sFail75.9sFail27.8sFail41.5sFail32.8sFail26.5s
debug-infinite-recursionDebuggingPass62.8sPass42.7sPass37.9sPass54.0sPass40.9sPass43.5sPass47.4s
debug-network-false-leadDebuggingFail212.9sFail193.1sFail240.0sFail182.5sFail105.0sFail82.8sFail130.7s
devshell-tooling-contractDev shellsPass46.9sPass131.1sPass53.2sPass39.0sPass60.5sPass56.8sPass51.0s
fetcher-source-pinFetchersPass96.8sPass138.4sPass107.0sPass36.9sPass53.8sPass26.1sPass23.0s
fhs-binary-wrapperPackagingPass98.0sPass68.5sPass142.7sPass47.1sPass51.1sPass47.6sPass43.7s
flake-input-package-selectionFlakesPass29.3sPass63.9sPass24.4sPass21.7sPass31.3sPass36.8sPass32.1s
flake-per-system-outputsFlakesPass212.4sPass184.7sFail240.0sPass74.9sPass74.1sPass53.9sPass87.6s
home-manager-extra-special-argsFlakesPass135.5sPass138.4sPass80.5sPass131.9sPass43.2sPass25.8sPass21.9s
home-manager-wsl-module-importModulesPass39.3sPass78.0sPass44.3sPass36.5sPass39.7sPass36.6sPass31.6s
home-manager-xdg-filesModulesPass45.7sPass49.1sPass56.9sPass33.1sPass34.3sPass25.9sPass28.1s
issue-report-qualityDebuggingFail38.8sFail50.2sFail39.1sFail89.3sFail49.1sFail36.8sFail28.7s
lang-attrsets-normalizeNix languagePass87.3sPass80.0sPass74.6sPass67.7sPass80.5sPass56.1sPass68.4s
module-path-compositionNix languagePass98.1sPass51.7sPass49.6sPass38.2sPass47.1sPass36.1sPass43.8s
module-service-optionsModulesFail124.8sPass106.6sPass84.6sPass81.2sFail64.1sFail41.0sFail67.8s
module-stale-option-migrationModulesPass41.5sPass37.7sPass63.1sPass29.9sPass38.3sPass37.3sPass19.7s
module-system-boundariesModulesPass55.6sPass81.8sPass40.8sPass45.3sPass41.1sPass40.6sPass29.8s
mutable-config-home-managerModulesPass82.8sPass42.8sPass63.3sPass38.2sPass44.2sPass41.2sFail46.7s
nushell-command-not-foundModulesNo data--No data--No data--No data--Fail50.4sFail60.0sFail44.1s
overlay-module-boundaryOverlaysPass69.1sPass58.4sPass95.8sPass34.2sPass58.4sPass29.0sPass34.2s
overlay-override-packageOverlaysPass44.0sPass70.9sPass50.5sPass48.6sPass53.9sFail32.8sPass61.2s
package-name-lookup-contractPackagingPass51.2sPass54.0sPass26.6sPass36.4sPass39.0sPass41.3sPass53.9s
package-python-applicationPackagingPass133.5sFail185.6sFail240.0sPass51.5sPass69.3sFail46.0sFail42.1s
package-stdenv-cliPackagingPass182.2sFail203.1sFail184.5sFail30.3sFail72.6sFail63.3sPass67.4s
purity-wrapper-derivationPurityPass145.0sPass107.3sPass79.1sPass100.4sPass39.0sPass38.0sFail52.5s
python-cuda-uv2nix-patchPackagingPass44.4sPass52.8sPass68.9sPass35.4sPass51.8sPass34.8sPass32.1s
rust-no-network-buildPackagingNo data--No data--No data--No data--Fail56.5sFail37.7sFail64.1s
string-escaping-systemdNix languagePass231.9sPass83.1sFail122.8sFail111.6sPass70.3sPass43.2sPass111.0s
xdg-portal-mergeModulesNo data--No data--No data--No data--Fail90.8sFail38.3sFail36.1s

Elapsed task time for the same fixed baseline runs.

Timing follows the exact columns above: xhigh Codex runs and the historical Claude default composite, each with a 240-second per-task timeout.

Charting 7 of 7 model columns

Run provenance (7)
  • GPT-5.526-task corpus · xhigh20260624T182835Z-4ad8b555 (+2)
  • GPT-5.426-task corpus · xhigh20260624T190640Z-fa04a19c (+2)
  • GPT-5.4 mini26-task corpus · xhigh20260624T194359Z-268b0abe (+2)
  • Claude Opus 4.826-task corpus · default20260624T202141Z-881ef1e9 (+2)
  • GPT-5.6 Sol29-task corpus · xhigh20260709T175817Z-cb86c575
  • GPT-5.6 Terra29-task corpus · xhigh20260709T175820Z-36a5f2f2
  • GPT-5.6 Luna29-task corpus · xhigh20260709T175826Z-b8e5f041

Several outcome patterns repeat across runs.

Native-container outcome

The xhigh/default baseline rows did not pass the native NixOS container task; GPT-5.5 low passed it in the effort sweep.

Debugging and reports

The false-lead debugging task and issue-report task were not passed by any row in this set.

Packaging variation

The Python application and stdenv CLI tasks show different pass patterns across models and effort levels.

String escaping

GPT-5.5 and GPT-5.4 passed the baseline string-escaping task; GPT-5.4 mini and Claude Opus 4.8 did not.