Decision date: 2026-10-03. This selects the new tasks to author from task-candidates-designed.md and task-candidates-community.md, following the recommendations in benchmark-design-research.md.
Selection rules
- Take the highest-ranked candidates whose trap was verified locally or is
documented in a primary source.
- Skip candidates whose trap overlaps an already selected one, so one bad
evaluator cannot sink several tasks at once.
- Bring every category to at least five active tasks where candidates allow.
- Prompts are symptom-driven: observed behaviour, failing command, the
interface the evaluator will call. No source links, no recipe.
- Evaluators use the vendored nixpkgs
lib($NIXBENCH_VENDOR_LIB) and
real lib.evalModules or real fixed points wherever the trap involves the module system or overlays. Every criterion has a rejecting fixture and at least one passing alternative that differs from the reference.
Selected: designed candidates (16)
| Id | Category | Difficulty |
|---|---|---|
debug-freeform-config-cycle | debugging | hard |
module-deferred-schema-default | modules | hard |
overlay-finalattrs-reoverride | overlays | hard |
scope-override-transitive-dependencies | overlays | hard |
lazy-selected-report-validation | nix-language | hard |
debug-import-argument-cycle | debugging | hard |
module-priority-submodule-apply | modules | hard |
flake-nested-follows-identity | flakes | hard |
string-command-dependency-context | nix-language | hard |
overlay-composed-final-prev | overlays | hard |
source-filter-traversal-stability | flakes | hard |
module-migration-assertion-gate | modules | hard |
lazy-recursive-update-closure | nix-language | hard |
fetcher-fixed-output-identity | fetchers | hard |
lazy-type-error-boundary | nix-language | hard |
purity-explicit-release-inputs | purity | medium |
Skipped: debug-contextful-attribute-name (overlaps the string-context task), string-shell-template-roundtrip and string-literal-replacement-order (reserve), flake-host-system-output-boundary (reserve), purity-sandbox-phase-confusion (reserve).
Selected: community candidates (8)
| Id | Category | Difficulty |
|---|---|---|
argument-default-forwarding | nix-language | easy |
mkforce-across-option-reexport | modules | medium |
module-generated-jobs-fixed-point | modules | medium |
python-build-backend-false-lead | debugging | medium |
follows-preserve-python-compatibility | flakes | medium |
runtime-tools-outside-devshell | devshells | medium |
package-offline-test-selection | packages | medium |
editor-project-state-isolation | devshells | medium |
Skipped because they overlap a designed task: context-preserving-trim, fetcher-stale-fixed-output-cache, source-filter-stable-store-name, module-import-source-before-pkgs. Skipped because the evaluator cannot be made deterministic offline within 60 seconds: nested-npm-dependency-rebuild, nuget-sdk-dependency-reconciliation, fhs-build-output-staging, gc-generations-shared-inodes, home-manager-fish-state-exclusion, samba-runtime-secret-provisioning. Remaining candidates stay in the pool for a later minor release.
Existing tasks
Calibration on the 2.0 digest showed 27 of 29 tasks solved by every configuration. Those tasks move to the deprecated lifecycle state with the reason saturated and stay in the repository for regression use. They do not enter 3.0 active scoring. rust-no-network-build (0% solve rate) and devshell-tooling-contract (mixed) stay active. Tasks whose evaluators were rewritten for semantic grading in 3.0 (module-service-options, string-escaping-systemd, module-system-boundaries, xdg-portal-merge) re-enter as calibrating because their digests changed and the old solve rate no longer describes them.
Resulting active corpus (target)
| Category | Active tasks |
|---|---|
| debugging | 3 |
| devshells | 3 |
| fetchers | 1 |
| flakes | 3 |
| modules | 8 |
| nix-language | 6 |
| overlays | 3 |
| packages | 2 |
| purity | 1 |
Thirty tasks. debugging, devshells, fetchers, flakes, overlays, packages, and purity remain below the five-task threshold and are reported descriptively only.