2026-06-24 through 2026-07-10 Model Comparison Runs

These are local NixBench runs against the corpus version current on each run date. Historical rows cover 26 tasks; the July 9-10 GPT-5.6 rows cover the expanded 29-task corpus and retain their own denominator in every score.

The raw results/ artifacts are intentionally not tracked because they contain logs, temporary diffs, and machine-local paths. This document records the durable statistics needed for comparison.

The first comparison on June 24 covered the then-current 24-task corpus. On June 25, home-manager-extra-special-args and overlay-module-boundary were added to the displayed comparison by running those two tasks separately for each baseline model. Rows marked (+2) combine the original 24-task run with those two supplemental task artifacts.

On June 27, gpt-5.4-mini was run across the missing low, medium, and high Codex reasoning effort settings.

On June 27 and June 28, Claude Opus 4.8 was run with explicit Claude Code effort settings.

On July 9, GPT-5.6 Sol, Terra, and Luna were each run at xhigh effort against all 29 current tasks using the Codex CLI bundled with the desktop app.

On July 10, the GPT-5.6 effort sweep added low, medium, and high rows for all three models, plus max rows for Sol and Luna. Sol ultra and Terra max/ultra were paused before they started and are not represented below.

Summary

All runs used a 240 second per-task agent timeout and the same task prompt contract: read NIXBENCH_PROMPT.md, edit only the copied task workdir, do not inspect hidden evaluators, run local checks if useful, then stop.

Effort Sweep Notes

Baseline Per-Task Results

This table shows the xhigh/default baseline columns used by the detailed results page.

The Codex columns use the original xhigh baselines. The Claude column preserves the original default composite from 20260624T202141Z-881ef1e9 plus the two supplemental task artifacts; it is distinct from the later explicit Claude xhigh run.

Outcome Notes

Commands

Effort sweep Codex runs used this shape, with <model> set to gpt-5.5 or gpt-5.4 and <effort> set to low, medium, or high:

python3 -u bench.py run-all --agent-timeout-seconds 240 --agent-cmd 'codex -c model_reasoning_effort=<effort> --ask-for-approval never exec -m <model> --ephemeral --skip-git-repo-check --sandbox workspace-write "You are in a temporary NixBench benchmark task workspace. Read NIXBENCH_PROMPT.md, then edit the local starter files to satisfy it. Only modify files in this directory. Do not inspect hidden evaluator files or the original task directory. Run local checks if useful, then stop."'

The original xhigh/default comparison commands did not pass -c model_reasoning_effort=...; the local Codex config was set to model_reasoning_effort = "xhigh" for the Codex baseline runs.

The June 27 gpt-5.4-mini effort runs used the same command shape with <model> set to gpt-5.4-mini.

The Claude Opus effort runs used the same Claude Code command shape with --model opus and --effort low, --effort high, or --effort xhigh.

The July 9 GPT-5.6 runs used the compatible Codex binary bundled with the desktop app because the separately installed Codex CLI 0.143.0 rejected these models as requiring a newer version. Each run used the same prompt contract and timeout shown above, with <model> set to gpt-5.6-sol, gpt-5.6-terra, or gpt-5.6-luna, and explicit model_reasoning_effort="xhigh".

The July 10 GPT-5.6 effort rows used that same bundled binary and command shape with explicit low, medium, high, or max effort. The first attempt reached the account usage limit after eight complete rows; quota-contaminated partial directories were discarded and the affected rows were rerun from task 1 after reset. Every recorded summary above is a complete, quota-clean 29-task run.