Task corpus
21 tasks, 5 categories
Each task is a small broken Nix repository, a prompt describing the symptom, and a hidden evaluator with named criteria. A task passes when every required criterion passes. Optional criteria are diagnostics only.
- easy (author)
- 1
- medium (author)
- 1
- hard (author)
- 19
devshells1 task · reported descriptively
Development Shell Tooling Contract
devshell-tooling-contract
easycalibratingshell.nix is still a placeholder that returns {}, so there is no development shell to enter. Turn it into the project's development shell.
1 starter fileevaluator timeout 60s
4 criteria · edits shell.nix
- shell-constructionevaluation · 25 pts
- required-toolswrong-value · 25 pts
- formatter-contractwrong-value · 25 pts
- experimental-featureswrong-value · 25 pts
modules15 tasks
Generate Units From A Module-Defined Job List
module-generated-jobs-fixed-point
mediumcalibratingBackup units used to be written inline for a single job, and that evaluated.
5 starter filesevaluator timeout 60s
5 criteria
- option-jobs-evaluateevaluation · 25 pts
- unit-nameswrong-value · 20 pts
- unit-fieldswrong-value · 20 pts
- disabled-jobswrong-value · 15 pts
- coexists-with-other-moduleswrong-value · 20 pts
Run A Per-Boot Spool Preparation At The Right Point Of Boot
boot-hook-marker-survives-reboot
hardcalibratingThis repository configures the upload ingest nodes and their bastion (see README.md). For the 26.05 pin the spool's per-boot preparation was moved to a systemd unit in the initrd (docs/CHANGELOG.md, 2026-10-02).
20 starter filesevaluator timeout 600s
5 criteria
- ingestd-starts-first-bootwrong-value · 25 pts
- prepare-once-per-bootwrong-value · 20 pts
- marker-after-rebootwrong-value · 25 pts
- spool-on-volumewrong-value · 15 pts
- bastion-unchangedwrong-value · 15 pts
Migrate Two Mail Hosts On Different Dovecot Releases To Structured Settings
dovecot-structured-settings-versions
hardcalibratingSince the nixpkgs pin was updated, the deploy pipeline rejects both mail hosts.
19 starter filesevaluator timeout 120s
5 criteria
- hosts-evaluateevaluation · 15 pts
- format-versionswrong-value · 20 pts
- tls-pathswrong-value · 20 pts
- protocols-and-authwrong-value · 20 pts
- mail-storage-preservedwrong-value · 25 pts
ESPHome Forgets Its Devices On Every Reboot
esphome-persistence-wrong-state-path
hardcalibratingThis repository configures our home hosts (see README.md). Every host has a tmpfs root and keeps state through fleet.persist (docs/persistence.md).
19 starter filesevaluator timeout 120s
6 criteria
- hosts-evaluateevaluation · 15 pts
- static-service-userwrong-value · 15 pts
- state-persistedwrong-value · 20 pts
- state-ownershipwrong-value · 15 pts
- sandbox-paths-keptwrong-value · 20 pts
- other-persistence-unchangedwrong-value · 15 pts
Restore Routing Through A Gateway That Silently Stopped Forwarding
firewall-nftables-forwarding-silently-dropped
hardcalibratingThis repository configures the branch-office gateway and one host on each side of it (see README.md). After the last round of changes (docs/CHANGELOG.md, 2026-10-02) every host still evaluates and builds, but the deploy pipeline's VM test now fails and the rollout is blocked.
17 starter filesevaluator timeout 600s
5 criteria
- lan-reaches-serverswrong-value · 25 pts
- blocked-ports-stay-blockedwrong-value · 20 pts
- servers-cannot-open-lanwrong-value · 15 pts
- firewall-policy-keptwrong-value · 20 pts
- policy-survives-firewall-reloadwrong-value · 20 pts
Restore A Fleet Security Baseline That Silently Disappeared
hardened-profile-import-no-op
hardnew, not yet in lifecycleThis repository configures three hosts (see README.md). Last month we moved them to the current nixpkgs pin. All three evaluated cleanly, deployed without errors and have run normally since. This week's quarterly baseline audit filed findings on every host.
23 starter filesevaluator timeout 120s
5 criteria
- hosts-evaluateevaluation · 10 pts
- kernel-hardeningwrong-value · 25 pts
- network-hardeningwrong-value · 20 pts
- userns-policywrong-value · 20 pts
- firewall-scopedwrong-value · 25 pts
Move A Stage-1 Storage Hook To The Systemd Initrd
initrd-hook-systemd-stage1-migration
hardcalibratingThis repository configures two storage appliances and an edge router (see README.md).
21 starter filesevaluator timeout 120s
6 criteria
- hosts-evaluateevaluation · 20 pts
- systemd-stage1-keptwrong-value · 15 pts
- hook-orderingwrong-value · 25 pts
- hook-runs-in-initrdunavailable-helper · 15 pts
- root-on-mapperwrong-value · 10 pts
- stage2-and-edge-preservedwrong-value · 15 pts
Vendor Relay Never Starts After The nixpkgs Update
service-requires-removed-network-setup
hardnew, not yet in lifecycleThis repository configures our log gateways (see README.md). Last night's deploy moved every host to the current nixpkgs pin and also shipped the vendor drop-box from OPS-31 (docs/vendor-dropbox.md) to ingest.
15 starter filesevaluator timeout 120s
6 criteria
- hosts-evaluateevaluation · 15 pts
- relay-requires-addresswrong-value · 20 pts
- relay-ordered-after-addresswrong-value · 20 pts
- vendor-link-intactwrong-value · 15 pts
- vendor-password-loginwrong-value · 15 pts
- sibling-ssh-unchangedwrong-value · 15 pts
Bring Two Stalwart Hosts Through The Module State Change
stalwart-state-version-migration
hardcalibratingThis repository configures our two mail exchangers, mx1 and mx2 (see README.md and docs/hosts.md).
18 starter filesevaluator timeout 120s
6 criteria
- hosts-evaluateevaluation · 15 pts
- legacy-store-preservedwrong-value · 25 pts
- fresh-host-defaultswrong-value · 20 pts
- unit-overrides-appliedwrong-value · 15 pts
- secrets-and-backup-followwrong-value · 15 pts
- listeners-unchangedwrong-value · 10 pts
Keep A Sandboxed Daemon Writing Its State And Socket
state-directory-ownership-after-hardening
hardcalibratingThis repository configures two ledger hosts (see README.md). Since the security-review follow-up (docs/CHANGELOG.md, 2026-09-30) both hosts still evaluate and build, but the deploy pipeline refuses to roll out ledger-a: in its VM boot test ledgerd never comes up.
18 starter filesevaluator timeout 600s
6 criteria
- ledgerd-activewrong-value · 15 pts
- state-owner-and-modewrong-value · 25 pts
- socket-createdwrong-value · 15 pts
- hardening-keptwrong-value · 20 pts
- sibling-service-unaffectedwrong-value · 10 pts
- standby-host-bootswrong-value · 15 pts
Bring A Unix-Socket Onion Service Back After The nixpkgs Update
tor-onion-socket-outside-chroot
hardnew, not yet in lifecycleThis repository configures our two onion-service hosts (see README.md). After we moved to the current nixpkgs pin, both hosts evaluated cleanly and deployed without errors.
22 starter filesevaluator timeout 120s
5 criteria
- hosts-evaluateevaluation · 10 pts
- tor-binds-socket-dirwrong-value · 30 pts
- socket-dir-lifecyclewrong-value · 25 pts
- tor-after-backendwrong-value · 15 pts
- mirror-unchangedwrong-value · 20 pts
Scanner FTP Logins Fail After The nixpkgs Update
vsftpd-local-login-without-pam
hardcalibratingThis repository configures our office servers (see README.md). Yesterday's deploy moved every host to the current nixpkgs pin. All three hosts evaluated and switched without an error or a warning, but since then none of the scanners can upload to files: every FTP login is…
18 starter filesevaluator timeout 120s
5 criteria
- hosts-evaluateevaluation · 15 pts
- ftp-pam-policywrong-value · 30 pts
- system-password-authwrong-value · 15 pts
- user-restrictions-keptwrong-value · 20 pts
- mirror-unchangedwrong-value · 20 pts
Get An EAP-TLS Kiosk Back On Wi-Fi After The nixpkgs Update
wireless-eap-key-unreadable-after-hardening
hardcalibratingThis repository configures our two kiosks (see README.md). We moved them to the current nixpkgs pin. Both hosts evaluated cleanly and fleet-deploy switched both without errors. kiosk-warehouse is fine. kiosk-lobby has not been back on the network since its next reboot.
18 starter filesevaluator timeout 120s
5 criteria
- hosts-evaluateevaluation · 10 pts
- eap-configwrong-value · 25 pts
- key-provisionedwrong-value · 25 pts
- daemon-hardenedwrong-value · 20 pts
- psk-host-unchangedwrong-value · 20 pts
Move A Mesh Node Key Out Of The Store Without Changing Its Identity
yggdrasil-credential-key-owner
hardcalibratingAfter moving the fleet to the new nixpkgs pin, the pre-deploy check fails for edge.
23 starter filesevaluator timeout 120s
5 criteria
- edge-evaluatesevaluation · 15 pts
- edge-key-from-runtime-secretwrong-value · 25 pts
- dev-keeps-persistent-keyswrong-value · 15 pts
- key-never-in-storeimpurity · 20 pts
- mesh-settings-preservedwrong-value · 25 pts
nix-language1 task · reported descriptively
Reject Malformed Records Without Aborting The Batch
lazy-type-error-boundary
hardcalibratingnix eval --json -f validate-example.nix should print one result per record. Records whose count is unavailable (throw) are already reported as invalid, but the cache record, whose offset is a string, aborts the whole batch with error: cannot add a string to an integer, and no…
2 starter filesevaluator timeout 60s
5 criteria · edits validate.nix
- valid-records-computedwrong-value · 20 pts
- type-mismatch-rejectedevaluation · 25 pts
- missing-fields-rejectedmissing-attr · 15 pts
- thrown-values-rejectedevaluation · 20 pts
- disabled-records-unforcedevaluation · 20 pts
overlays2 tasks · reported descriptively
Keep A Package Report Correct Across Repeated Overrides
overlay-finalattrs-reoverride
hardcalibratingEdit package.nix and overlay.nix. pkgs.nix is a small stand-in for nixpkgs. It uses the real nixpkgs lib and a stdenv.mkDerivation that behaves like the nixpkgs one, including finalAttrs and overrideAttrs with an attrset, a one-argument function, or a two-argument function.
5 starter filesevaluator timeout 60s
5 criteria
- vendor-overlay-resultwrong-value · 20 pts
- version-reoverridewrong-value · 20 pts
- dependency-reoverridewrong-value · 20 pts
- passthru-preservedmissing-attr · 20 pts
- consumers-isolatedwrong-value · 20 pts
Compose Python Library Pins Across Team And Host Overlays
python-package-set-override-fleet
hardcalibratingThis repository packages the fleet's Python services against the pinned nixpkgs on NIX_PATH. The README explains how CI evaluates it.
39 starter filesevaluator timeout 120s
6 criteria
- apps-instantiateevaluation · 15 pts
- click-pin-resolves-oncewrong-value · 20 pts
- msgpack-pin-resolves-oncewrong-value · 15 pts
- edge-compression-dependencywrong-value · 15 pts
- host-overlay-repin-winswrong-value · 15 pts
- app-interfaces-unchangedoverfit · 20 pts
packages2 tasks · reported descriptively
Keep Rust Builds Offline
rust-no-network-build
hardcalibratingvision-indexer fails to build in the Nix sandbox. With the download-binaries Cargo feature, its build scripts try to download a prebuilt ONNX Runtime (ORT_STRATEGY = "download"), and they fetch the model file from the MODEL_URL set in the starter's env.
1 starter fileevaluator timeout 60s
4 criteria · edits package.nix
- package-sourcewrong-value · 25 pts
- system-onnxruntimeimpurity · 25 pts
- pinned-modelimpurity · 25 pts
- no-network-escapeimpurity · 25 pts
Cross-Compile A C CLI With Build-Time Tools
stdenv-cross-and-host-tools
hardcalibratingThis is the logpipe repository with its Nix packaging; the README shows how it is evaluated against the pinned nixpkgs on NIX_PATH.
14 starter filesevaluator timeout 120s
5 criteria
- instantiatesevaluation · 15 pts
- pkg-config-and-librarieswrong-value · 20 pts
- generator-build-compilerwrong-value · 20 pts
- man-page-from-scdocwrong-value · 25 pts
- wrapper-host-zstdwrong-value · 20 pts
56 archived tasks from earlier corpus generations
Archived tasks live under archive/<generation>/tasks. They are not scored by the current corpus; most were retired because nearly every configuration solved them. They stay runnable for regression use.
3.1 corpus 9 tasksNixBench 3.1.0 results
Find The Recursion Behind A Fleet-Wide Rebuild Failure
debug-infinite-recursion-fleet · debugging · hard
Per-Project Editor State Isolation
editor-project-state-isolation · devshells · medium
Make cgit And Smart-HTTP Clones Agree On Which Repositories Are Published
cgit-export-policy-agreement · modules · medium
Retire Lovelace Top-Level Mode Without Handing Dashboards To The UI
home-assistant-declarative-dashboards · modules · medium
Split Fleet Sync Between NixOS And Home Manager
home-manager-nixos-module-boundary-fleet · modules · hard
Web Fleet After The 26.05 Bump: nginx TLS And ACME
nginx-acme-virtualhost-migration · modules · hard
Make The Ingest Worker Start Reliably On Both Hosts
systemd-unit-ordering-and-paths · modules · hard
Persisting A Service That Left DynamicUser On A tmpfs Root
users-groups-impermanence-boundary · modules · hard
Vendor A Rust Workspace Member With Git Dependencies
rust-workspace-vendored-deps · packages · hard
3.0 corpus 24 tasksNixBench 3.0.0 results
Debug A Freeform Module Recursion
debug-freeform-config-cycle · debugging · hard
Debug A Module Import That Depends On Module Arguments
debug-import-argument-cycle · debugging · hard
Debug A Python Build Failure Blamed On The Interpreter
python-build-backend-false-lead · debugging · medium
Runtime Tools Outside the Development Shell
runtime-tools-outside-devshell · devshells · medium
Fixed-Output Source Identity After a Release Bump
fetcher-fixed-output-identity · fetchers · hard
Share Nixpkgs Through a Nested Flake Input Graph
flake-nested-follows-identity · flakes · hard
Deduplicate Flake Inputs Without Breaking a Python Tool
follows-preserve-python-compatibility · flakes · medium
Filter a Flake Source Without Losing Files or Identity
source-filter-traversal-stability · flakes · hard
Force a Value Where the Forwarded Option Conflicts
mkforce-across-option-reexport · modules · medium
Keep Plugin Schemas Through Shared Customization
module-deferred-schema-default · modules · hard
Migrate Renamed and Removed Options Behind an Assertion Gate
module-migration-assertion-gate · modules · hard
Repair Listener Priorities, Conditions, and Normalization in Submodules
module-priority-submodule-apply · modules · hard
NixOS Service Module Options
module-service-options · modules · hard
Respect NixOS, Home Manager, And nix-darwin Boundaries
module-system-boundaries · modules · hard
Merge XDG Portal Configuration
xdg-portal-merge · modules · hard
Forward Wrapper Defaults To The Builder
argument-default-forwarding · nix-language · easy
Recompute Derived Release Fields After Updates
lazy-recursive-update-closure · nix-language · hard
Validate Only The Exported Projection Of Enabled Records
lazy-selected-report-validation · nix-language · hard
Keep Command Dependencies Without Copying Display Paths
string-command-dependency-context · nix-language · hard
Escape Shell Variables In Nix Strings
string-escaping-systemd · nix-language · medium
Make A Vendor Overlay Compose With Earlier And Later Overlays
overlay-composed-final-prev · overlays · hard
Make A Scope Override Reach Transitive Consumers
scope-override-transitive-dependencies · overlays · hard
Run A Package's Offline Tests In The Sandbox
package-offline-test-selection · packages · medium
Release Manifest From Explicit Inputs
purity-explicit-release-inputs · purity · medium
2.0 corpus 23 tasksNixBench 2.0 results
Debug Infinite Recursion
debug-infinite-recursion · debugging · easy
1 starter file60sprompt
Debug Network Symptoms Without Chasing False Leads
debug-network-false-lead · debugging · medium
1 starter file60sprompt
Write A Verifiable Nixpkgs Issue Report
issue-report-quality · debugging · medium
1 starter file60sprompt
Pin A GitHub Source Fetcher
fetcher-source-pin · fetchers · easy
1 starter file60sprompt
Select The Right Flake Package Output
flake-input-package-selection · flakes · medium
1 starter file60sprompt
Per-System Flake Outputs
flake-per-system-outputs · flakes · medium
1 starter file60sprompt
Pass Flake Inputs Into Home Manager
home-manager-extra-special-args · flakes · medium
2 starter files60sprompt
Choose Native NixOS Containers
container-native-vs-oci · modules · medium
1 starter file60sprompt
Import Home Manager As A NixOS Module On WSL
home-manager-wsl-module-import · modules · medium
1 starter file60sprompt
Manage Home Files Declaratively
home-manager-xdg-files · modules · easy
1 starter file60sprompt
Migrate Stale NixOS Options
module-stale-option-migration · modules · easy
1 starter file60sprompt
Separate Mutable App State From Home Manager Defaults
mutable-config-home-manager · modules · medium
1 starter file60sprompt
Wire Command-Not-Found Into Nushell
nushell-command-not-found · modules · medium
1 starter file60sprompt
Normalize Package Attribute Sets
lang-attrsets-normalize · nix-language · easy
1 starter file60sprompt
Compose Module Paths From Arguments
module-path-composition · nix-language · medium
1 starter file60sprompt
Keep Overlays And Module Options Separate
overlay-module-boundary · overlays · medium
2 starter files60sprompt
Overlay Package Override
overlay-override-package · overlays · hard
2 starter files60sprompt
Wrap FHS-Oriented Binaries
fhs-binary-wrapper · packages · medium
1 starter file60sprompt
Use Existing Package Names
package-name-lookup-contract · packages · easy
1 starter file60sprompt
Package A Python Application
package-python-application · packages · medium
1 starter file60sprompt
Package A stdenv CLI
package-stdenv-cli · packages · medium
1 starter file60sprompt
Patch Python CUDA Package Inputs
python-cuda-uv2nix-patch · packages · hard
1 starter file60sprompt
Pure Wrapper Derivation
purity-wrapper-derivation · purity · medium
1 starter file60sprompt