NixBench 3.0 task candidates designed from semantics

Research date: 2026-10-03. These are 21 ranked task designs, not activated corpus tasks or measured model failures. Every Nix trap below was evaluated locally with Nix 2.34.8. The ranking estimates relative discrimination among these candidates. It does not assign empirical solve rates or claim that frontier agents have failed them.

Evidence and constraints

I read README.md, task-format.md, and benchmark-design.md, then every file in both overlay-override-package and module-system-boundaries, including their metadata, prompts, starter files, reference files, evaluators, and all associated overlay contracts and module contracts.

The overlay evaluator models only a one-argument attrs update. The module-boundary evaluator calls module functions directly and selects .config; it does not invoke the Nix module system. Their contracts correctly accept useful alternatives such as reordered packages and module attrsets, but they cannot exercise real priority merging, module import phases, scope fixed points, or two-argument finalAttrs behavior. The new designs target those gaps without turning syntax preferences into correctness requirements.

The locally available nixpkgs revision is d2ed99647a4b195f0bcc440f76edfa10aeb3b743, channel label nixos-25.05.810061.d2ed99647a4b. I copied its lib/ to /tmp/nixbench-candidate-research/lib, approximately 2 MiB. nix hash path returned sha256-ADZAgQwDYEM3eBGQnd0sYKg1xdnLPV91tve41H1hO5c=. This is the exact library tested, not a claim about every nixpkgs revision. A task author must vendor this pinned library and its license into the corpus and expose the same library to the agent. No evaluator should import a channel, fetch nixpkgs, or depend on an existing nixpkgs store path.

All probes ran offline. Most use the command below, with the named expression saved as <probe>.nix beside lib/:

nix eval --offline --option allow-import-from-derivation false --json \
  --file /tmp/nixbench-candidate-research/<probe>.nix

Pure-mode probes use nix eval --offline --json --option pure-eval true --expr '<expression>'. Reading a host-side expression file through --file under pure mode can fail before the candidate runs. For an eventual pure evaluator, first admit the candidate tree as a trusted store input, then import that store path. The local-flake probes use actual local path inputs, generate locks in scratch space, and run nix eval --offline --json --no-write-lock-file path:<fixture>#report.

Output blocks are actual stdout, or explicitly labeled excerpts of actual stderr. An exit code of 1 in a wrong probe is expected evidence. Store hashes and scratch paths in the transcripts are observations, not values to hard-code into an evaluator. No derivation was built and no remote input was fetched. Native stdenv probes used the already installed checkout only to corroborate the small proposed override fixture; the candidate evaluator would use vendored lib and the disclosed fixture. Native stdenv's metadata/build shaping is outside that fixture's contract.

Evaluator design shared by all candidates

Each candidate should become a small editable project with a symptom-driven prompt, one public local reproducer, and a stable constructor or module interface. Put that interface and all promised variant behavior in the public prompt; hidden probes may vary values and composition, not invent requirements. Keep the trusted evaluator and input mutations outside the candidate tree. Probe candidate functions through arguments, or copy candidates into evaluator-owned scratch space for local-flake input fixtures. Do not rewrite the candidate's implementation into the expected solution during testing.

Use the current task format: controlled category, difficulty hard or medium, timeout 60, systems = ["any"] for these host-independent projections, and four to eight required behavioral criteria totaling 100 points. A dummy derivation's system = "x86_64-linux" does not require an x86 host when it is never built. Write the complete schema-version-2 boolean criterion payload even after candidate parse/type/evaluation failures, then exit 1. Reserve exit 2+ for missing infrastructure. tryEval (deepSeq ...) is useful for catchable failures but cannot replace subprocess handling of parser/type errors or recursion. Use bounded per-probe processes and a total evaluator budget below 60 seconds; do not score interpreter crashes as infrastructure errors.

Every criterion needs a rejecting contract mutation and important passing alternatives. Include expected complete criterion vectors, not just one failing boolean. Run references and contracts twice for determinism. Compare semantic values, contexts, graph identities, or normalized byte outputs; avoid source regexes, callback-count requirements, whole-package deep forcing, and reference-output snapshots. A negative fixture must fail for its intended reason, not merely because a fixture path is absent. Keep poison thunks in unobserved fields to test laziness, but avoid unrestricted cyclic values in a fixture intended to finish normally.

The experiments show generous timing headroom, but they are small semantic probes, not finished 60-second evaluators. Budget roughly 5 seconds for each Nix subprocess and at most 10 probes per task, with a total deadline around 50 seconds. Batch independent observations when doing so does not let a fatal error erase all partial criteria. Record timeouts separately under the harness timeout protocol. Do not disguise a timeout as an ordinary semantic rejection and then count it as a healthy starter result. Validate references, starters, and passing/rejecting contracts before calibration.

The table deliberately includes three lower-confidence reserve tasks. Start calibration with ranks 1 through 16, use several trials across at least two correctness configurations, and retain only tasks with useful mixed outcomes. Follow the repository's leave-one-task-out point-biserial and minimum-observation rules rather than assigning discrimination from these author estimates. The two string-context tasks overlap and may prove redundant. The module tasks also need a correlation check.

Ranked candidates

Coverage is balanced by primary area: three laziness tasks, three overlay/scope tasks, three module tasks, three string tasks, three path/flake tasks, three purity tasks, and three evidence-led debugging tasks. Debugging candidates also deepen module and string-context coverage.

1. debug-freeform-config-cycle

Category: debugging. Difficulty: hard.

The freeform merger must discover and type-check its contents before returning config. A freeform message that reads config.mode recurses even when mode has a declared option. In this pinned library, replacing mkIf with a value-level if still recurses. Declaring both the controlling option and the computed option breaks the cycle while leaving unrelated freeform keys extensible.

Symptom-driven prompt sketch

nix eval --file demo.nix ends at modules.nix with infinite recursion encountered. The service worked until its status message began depending on mode. service.nix also has a renamed setting and a disabled plugin, but the supplied trace comes from the status computation. Repair the service so its message tracks each instance's mode, callers can override the message, and existing arbitrary string settings keep working. Keep the named instances independent.

Evaluator and alternate-input probes

Use real evalModules and place the repaired service schema in attrsOf (submodule ...). Vary instance names, on/off modes, empty instances, ordinary and mkForce message overrides, and unknown freeform string keys. Assert an absent instance stays absent, an empty instance receives defaults, and one instance's mode cannot change another's message. Invalid integer freeform data must still fail its declared string type. Run a second check with only the status computation changed back to a config-dependent freeform field; it must reproduce recursion. The decoy renamed setting and disabled plugin need passing contract fixtures so deleting them cannot count as repair.

Valid alternatives

Accept declared computed options, a nested declared settings schema with a freeform remainder, or another equivalent module layout that retains the public config paths and overrides. Accept module attrsets or functions. Do not require a particular mkIf spelling. Reject replacing the freeform type with unspecified just to suppress type checks.

Expected discrimination

High relative potential. The common advice to replace if with mkIf is insufficient here, and the trace points into the module implementation. An agent that reduces the cycle and tests the real library should solve it; there are no model trials yet.

Primary implementation references: [M], [T].

#### Probe freeform-wrong

let lib = import ./lib; in
(lib.evalModules { modules = [ ({ config, ... }: {
 freeformType = lib.types.attrsOf lib.types.str;
 config = lib.mkMerge [ { mode = "on"; }
   (lib.mkIf (config.mode == "on") { message = "ready"; }) ];
}) ]; }).config.message

Observed exit 1 in 0.0230 seconds.

Stderr excerpt:

       error: infinite recursion encountered
       at /tmp/nixbench-candidate-research/lib/modules.nix:257:21:
          256|                     options
          257|                     config
             |                     ^
          258|                     specialArgs

#### Probe freeform-right

let lib = import ./lib; in
(lib.evalModules { modules = [ ({ config, ... }: {
 freeformType = lib.types.attrsOf lib.types.str;
 options.mode = lib.mkOption { type = lib.types.str; default = "on"; };
 config.message = lib.mkIf (config.mode == "on") "ready";
}) ]; }).config.message

Observed exit 1 in 0.0240 seconds.

Stderr excerpt:

       error: infinite recursion encountered
       at /tmp/nixbench-candidate-research/lib/modules.nix:257:21:
          256|                     options
          257|                     config
             |                     ^
          258|                     specialArgs

#### Probe freeform-conditional-value

let lib = import ./lib; in
(lib.evalModules { modules = [ ({ config, ... }: {
 freeformType = lib.types.attrsOf lib.types.str;
 options.mode = lib.mkOption { type = lib.types.str; default = "on"; };
 config.message = if config.mode == "on" then "ready" else "idle";
}) ]; }).config.message

Observed exit 1 in 0.0246 seconds.

Stderr excerpt:

       error: infinite recursion encountered
       at /tmp/nixbench-candidate-research/lib/modules.nix:257:21:
          256|                     options
          257|                     config
             |                     ^
          258|                     specialArgs

#### Probe freeform-declared

let lib = import ./lib; in
(lib.evalModules { modules = [ ({ config, ... }: {
 freeformType = lib.types.attrsOf lib.types.str;
 options = {
   mode = lib.mkOption { type = lib.types.str; default = "on"; };
   message = lib.mkOption { type = lib.types.str; default = "idle"; };
 };
 config.message = lib.mkIf (config.mode == "on") "ready";
}) ]; }).config.message

Observed exit 0 in 0.0248 seconds.

Stdout:

"ready"

#### Probe submodule-defaults

let lib = import ./lib; in
let
 type = lib.types.attrsOf (lib.types.submodule ({ name, config, ... }: {
   options = {
     port = lib.mkOption { type = lib.types.int; default = 80; };
     label = lib.mkOption { type = lib.types.str; default = name; };
     url = lib.mkOption { type = lib.types.str; default = "${config.label}:${toString config.port}"; };
   };
 }));
 eval = defs: (lib.evalModules { modules = [
 { options.services = lib.mkOption { inherit type; default = {}; }; }
 ] ++ defs; }).config.services;
in {
 absent = eval [];
 empty = eval [ { services.api = {}; } ];
 merged = eval [ { services.api.port = lib.mkDefault 81; } { services.api.port = 82; } ];
}

Observed exit 0 in 0.0248 seconds.

Stdout:

{"absent":{},"empty":{"api":{"label":"api","port":80,"url":"api:80"}},"merged":{"api":{"label":"api","port":82,"url":"api:82"}}}

2. module-deferred-schema-default

Category: modules. Difficulty: hard.

A deferredModule stores modules for a later evaluation. An option default is still a low-priority definition of that module-valued option. Putting indispensable option declarations in that default means a caller definition can replace the schema before the inner evaluation. The wrong probe fails with port does not exist; adding the schema as a normal module contribution retains it.

Symptom-driven prompt sketch

The plugin runner evaluates with no customizations. Adding the documented { plugin.port = lib.mkDefault 81; } customization now says that port does not exist. Fix plugin-option.nix and its consumer so plugins remain modules, each instance gets its schema and defaults, and later instance configuration can override those defaults. The demo includes two instances whose ports must stay independent.

Evaluator and alternate-input probes

Evaluate the outer option and then feed its resulting module into a fresh inner evalModules. Probe no definitions, one module function, multiple merged module fragments, two independent instances, and port overrides at default, normal, and force priorities. Add an extra option declaration through a caller module and ensure it reaches the inner evaluator. Include a function that needs an inner specialArg to catch premature evaluation in the outer argument scope. A forced plugin definition is another required case if the schema is promised unconditionally; schema placement outside replaceable defaults must survive it.

Valid alternatives

Accept a deferredModule with static schema modules, schema imports in the consumer, or an equivalent explicit submodule boundary. A normal schema contribution shown in the probe fixes normal customizations but is insufficient for a forced replacement if unconditional schema retention is part of the final prompt. Do not accept evaluating the plugin early to an attrset of config values, since it loses declarations and later merging.

Expected discrimination

High relative potential. A plausible repair changes port's default or forwards more arguments but leaves the schema inside a replaceable outer default. This needs reasoning about two module evaluations and two priority levels.

The ordinary-contribution repair succeeds for normal definitions. The final probe uses deferredModuleWith.staticModules and also succeeds for a forced plugin replacement; it is the stronger reference direction for the unconditional-schema contract.

Primary implementation references: [M], [T].

#### Probe deferred

let lib = import ./lib; in
let
 outer = lib.evalModules { modules = [
 { options.plugin = lib.mkOption { type = lib.types.deferredModule;
     default = { options.port = lib.mkOption { type = lib.types.int; default = 80; }; }; }; }
 { plugin = { lib, ... }: { port = lib.mkDefault 81; }; }
 ]; };
 inner = lib.evalModules { modules = [ outer.config.plugin { port = 82; } ]; };
in inner.config.port

Observed exit 1 in 0.0242 seconds.

Stderr excerpt:

       error: The option `port' does not exist. Definition values:
       - In `<unknown-file>, via option plugin':
           {
             _type = "override";
             content = 81;
             priority = 1000;
           }

       It seems as if you're trying to declare an option by placing it into `config' rather than `options'!

#### Probe deferred-right

let lib = import ./lib; in
let
 outer = lib.evalModules { modules = [
 { options.plugin = lib.mkOption { type = lib.types.deferredModule;
     default = {}; };
   config.plugin = { options.port = lib.mkOption { type = lib.types.int; default = 80; }; }; }
 { plugin = { lib, ... }: { port = lib.mkDefault 81; }; }
 ]; };
 inner = lib.evalModules { modules = [ outer.config.plugin { port = 82; } ]; };
in inner.config.port

Observed exit 0 in 0.0236 seconds.

Stdout:

82

#### Probe deferred-static-force

let lib = import ./lib; in
let
 schema = { options.port = lib.mkOption { type = lib.types.int; default = 80; }; };
 eval = definition: let
   outer = lib.evalModules { modules = [
     { options.plugin = lib.mkOption {
         type = lib.types.deferredModuleWith { staticModules = [ schema ]; };
         default = {};
       }; }
     { plugin = definition; }
   ]; };
 in (lib.evalModules { modules = [ outer.config.plugin ]; }).config.port;
in { normal = eval { port = 81; }; forced = eval (lib.mkForce { port = 82; }); }

Observed exit 0 in 0.0201 seconds.

Stdout:

{"forced":82,"normal":81}

3. overlay-finalattrs-reoverride

Category: overlays. Difficulty: hard.

override changes a package function argument; overrideAttrs changes the builder attributes after that function has run. A passthru label closed over codec cannot be repaired by adding an attribute named codec. A finalAttrs reference follows later version overrides; a rec binding captures the original version. Replacing passthru loses unrelated keys, while conditional attribute names that inspect finalAttrs can recurse.

Symptom-driven prompt sketch

The vendor overlay reports codec v1 after the caller selects v2. Its release report also becomes stale when a downstream overlay bumps the version again. Repair package.nix and overlay.nix so the report describes the package actually produced, including later overrides, and downstream tooling can still read the existing passthru fields. The demo shows both the initial override and a second consumer override.

Evaluator and alternate-input probes

Use the small fixed-point fixture shown below plus real lib.makeOverridable, lib.extends and lib.toExtension. Probe opaque codec values, two later version changes, zero and multiple passthru keys, and both one-argument and two-argument override callbacks. Compare dependency selection, label, report, and preserved passthru values after every stage. Include a public finalPackage relationship if tested. Add contract mutations for a codec attribute override, rec-captured version, passthru replacement, finalAttrs-dependent attribute names, and a report based on previousAttrs instead of finalAttrs.

Valid alternatives

Accept package-function reapplication, override, or another function-level dependency injection that preserves later overrides. Accept one-argument callbacks where equivalent, and equivalent finalAttrs-derived computations. Do not check callback arity, override counts, variable names, or the source text of the fix. The fixture supports the specified callback/attribute behavior, not the full stdenv API.

Expected discrimination

High relative potential, but lower confidence than the real-module candidates. Three individually plausible fixes fail after the second override. The artificial builder limits realism; the actual stdenv corroboration reduces the risk of testing invented semantics.

The first probe is the proposed offline fixture. The next two are research corroboration against the locally installed real nixpkgs stdenv; they are not evaluator dependencies. The real constructor emitted version-without-src warnings, which are unrelated to the projection tested here. Only lib is needed for the final benchmark fixture. Do not claim that lib.makeOverridable or extendMkDerivation alone implements stdenv.overrideAttrs.

Primary implementation references: [F], [C], [D].

#### Probe override-fixture

let lib = import ./lib; in
let
 mk = input: let
   f = lib.toFunction input;
   attrs = f (attrs // { finalPackage = package; });
   package = attrs // (attrs.passthru or {}) // {
     overrideAttrs = change: mk (lib.extends (lib.toExtension change) f);
   };
 in package;
 base = lib.makeOverridable ({ codec ? "v1" }: mk (final: {
   version = "1";
   passthru = { inherit codec; keep = "present"; label = "${codec}:${final.version}"; };
 })) {};
 wrong = base.overrideAttrs { codec = "v2"; version = "2"; };
 right = (base.override { codec = "v2"; }).overrideAttrs (final: old: {
   version = "2";
   passthru = old.passthru // { report = final.passthru.label; };
 });
 again = right.overrideAttrs { version = "3"; };
in { wrong = wrong.passthru; right = right.passthru; again = again.passthru; }

Observed exit 0 in 0.0187 seconds.

Stdout:

{"again":{"codec":"v2","keep":"present","label":"v2:3","report":"v2:3"},"right":{"codec":"v2","keep":"present","label":"v2:2","report":"v2:2"},"wrong":{"codec":"v1","keep":"present","label":"v1:2"}}

#### Probe override-real-again

let
 pkgs = import /nix/var/nix/profiles/per-user/root/channels/nixos { system = "x86_64-linux"; };
 lib = pkgs.lib;
 package = lib.makeOverridable ({ codec ? "v1" }: pkgs.stdenv.mkDerivation (final: {
   pname = "demo"; version = "1"; dontUnpack = true;
   passthru = { inherit codec; label = "${codec}:${final.version}"; keep = "present"; };
 })) {};
 wrong = package.overrideAttrs (old: { codec = "v2"; version = "2"; });
 right = (package.override { codec = "v2"; }).overrideAttrs (final: old: {
   version = "2";
   passthru = old.passthru // { report = final.passthru.label; };
 });
  again = right.overrideAttrs { version = "3"; };
in { wrong = wrong.passthru; right = right.passthru; again = again.passthru; }

Observed exit 0 in 0.1882 seconds.

Stdout:

{"again":{"codec":"v2","keep":"present","label":"v2:3","report":"v2:3"},"right":{"codec":"v2","keep":"present","label":"v2:2","report":"v2:2"},"wrong":{"codec":"v1","keep":"present","label":"v1:2"}}

Stderr also contained the native stdenv version-without-src warning. The warning does not change these results.

#### Probe override-rec

let
 pkgs = import /nix/var/nix/profiles/per-user/root/channels/nixos { system = "x86_64-linux"; };
 p = pkgs.stdenv.mkDerivation rec { pname = "demo"; version = "1";
   passthru.label = "v${version}"; };
in (p.overrideAttrs { version = "2"; }).passthru.label

Observed exit 0 in 0.1584 seconds.

Stdout:

"v1"

#### Probe override-name-cycle

let lib = import ./lib; in
let
 mk = f: let attrs = f attrs; in attrs // {
   overrideAttrs = change: mk (lib.extends change f);
 };
 p = mk (_: { version = "1"; });
 bad = p.overrideAttrs (final: prev:
   lib.optionalAttrs (final.version == "2") { marker = true; }
   // { version = "2"; });
in bad.version

Observed exit 1 in 0.0181 seconds.

Stderr excerpt:

       error: infinite recursion encountered
       at /tmp/nixbench-candidate-research/override-name-cycle.nix:8:23:
            7|  bad = p.overrideAttrs (final: prev:
            8|    lib.optionalAttrs (final.version == "2") { marker = true; }
             |                       ^
            9|    // { version = "2"; });

#### Probe real-overlay-name-cycle

let
 pkgs = import /nix/var/nix/profiles/per-user/root/channels/nixos { system = "x86_64-linux"; };
 p = pkgs.stdenv.mkDerivation { pname = "demo"; version = "1"; };
 bad = p.overrideAttrs (final: prev:
   pkgs.lib.optionalAttrs (final.version == "2") { passthru.marker = true; }
   // { version = "2"; });
in bad.passthru.marker

Observed exit 1 in 0.1889 seconds.

Stderr excerpt:

       error: infinite recursion encountered
       at /nix/var/nix/profiles/per-user/root/channels/nixos/pkgs/stdenv/generic/make-derivation.nix:1:2806:

4. scope-override-transitive-dependencies

Category: overlays. Difficulty: hard.

A scope is a fixed point with its own callPackage. Updating its codec attribute with // does not rebuild the consumer closures. overrideScope extends the scope constructor and recomputes consumers against the new scope. Explicit callPackage arguments still outrank the automatically supplied scope arguments.

Symptom-driven prompt sketch

The package browser lists codec v2, but the application from the same scope still reports that it links v1. Fix scope.nix so a downstream scope override reaches every consumer. A separate diagnostic call intentionally injects a private codec and must keep that explicit choice. The unrelated package in the demo should keep its original settings.

Evaluator and alternate-input probes

Use real makeScope and callPackageWith. Add a codec-to-parser-to-app dependency chain, compose two overrideScope calls, and supply opaque attrsets as dependencies so string rewriting fails. Change a sibling dependency in a later extension. Probe explicit callPackage arguments and an added scope package. Compare values and dependency identities, not the number of calls or helper spellings.

Valid alternatives

Accept overrideScope, rebuilding the scope through its documented constructor, or another equivalent fixed-point construction. Preserve caller argument precedence. A deep attrset merge that leaves previously constructed app closures intact must fail.

Expected discrimination

High relative potential. Reading the final scope attrset gives convincing but incomplete evidence; the failure is in a transitive consumer. A single codec-only demo would be too easy, so downstream composition is essential.

Primary implementation references: [C], [F].

#### Probe scope

let lib = import ./lib; in
let
  s = lib.makeScope lib.callPackageWith (self: {
    codec = "v1";
    app = self.callPackage ({ codec }: { linked = codec; }) {};
  });
  bad = s // { codec = "v2"; };
  good = s.overrideScope (final: prev: { codec = "v2"; });
in { bad = bad.app.linked; good = good.app.linked;
     explicit = good.callPackage ({codec}: codec) { codec = "local"; }; }

Observed exit 0 in 0.0214 seconds.

Stdout:

{"bad":"v1","explicit":"local","good":"v2"}

5. lazy-selected-report-validation

Category: nix-language. Difficulty: hard.

tryEval and seq evaluate the outer shape, so an attrset containing a throwing field passes. deepSeq discovers that failure, but applying it to the entire input also forces fields that are deliberately irrelevant. The correct validation boundary is the exported projection of enabled records, after selection.

Symptom-driven prompt sketch

The report says a record is valid, then JSON export aborts with bad detail. A previous fix made an unrelated disabled record abort the entire report. Repair report.nix so enabled records are validated through the fields that the report exports, rejected records receive the documented failure result, and disabled records or private debug fields are never needed to produce the report.

Evaluator and alternate-input probes

Probe valid records, throws nested in exported lists/attrsets, a throwing private field, a disabled record with throwing payload, and reordered inputs. Require the same result after adding unrelated poison fields. Deeply force each produced report to catch delayed failure. Add a negative fixture using shallow tryEval and another deepSeq-ing all inputs. Specify that payload failures are catchable throw/assert failures; do not demand catching arbitrary interpreter errors or recursion.

Valid alternatives

Accept explicit projection before deepSeq, recursive validation of the documented exported shape, or equally selective traversal. Preserve public record ordering and failure representation. Do not require deepSeq when another implementation proves the same property.

Expected discrimination

High relative potential. The first obvious patch improves validation but violates laziness, while the second can accidentally stop validating nested fields. Hidden probes need both directions; otherwise this is a basic tryEval exercise.

Primary implementation references: [N].

#### Probe lazy-report

let
  payload = { answer = 42; detail = throw "bad detail"; };
  check = x: (builtins.tryEval x).success;
in {
  shallow = check payload;
  seqOnly = check (builtins.seq payload true);
  deep = check (builtins.deepSeq payload true);
  selected = check (builtins.deepSeq { inherit (payload) answer; } true);
}

Observed exit 0 in 0.0188 seconds.

Stdout:

{"deep":false,"selected":true,"seqOnly":true,"shallow":true}

6. debug-import-argument-cycle

Category: debugging. Difficulty: hard.

Module imports are resolved before config._module.args is available. Reading an argument supplied only through _module.args while constructing imports creates recursion. specialArgs is available during graph collection. Adding rec, renaming the argument, or moving its _module.args definition to another module cannot fix this phase dependency.

Symptom-driven prompt sketch

The trace ends at args.${name} or config._module.args.${name} while resolving featureModule. This started when a feature module became selectable by the caller. Repair entry.nix so the caller's feature module is imported and its ordinary runtime settings still receive their configured values. The repo contains a package overlay and a host-system selector, but the provided reproducer evaluates only this module entry point.

Evaluator and alternate-input probes

Run real evalModules with several feature module attrsets, functions, and local paths. Vary the option they declare and an ordinary argument consumed only in config. Probe a module whose own imports require the early argument to catch a repair that manually calls only the first function. Require changes to the feature module to affect output, and a disabled optional module to stay absent. Keep the runtime argument intentionally supplied through normal module configuration where that is the public interface.

Valid alternatives

Accept specialArgs, a lexical closure over the supplied module, or explicit top-level import wiring. Do not require moving every _module.args value into specialArgs. Accept equivalent graph construction as long as caller choice and later config merging survive.

Expected discrimination

High-to-moderate relative potential. The trace has a precise clue, so agents familiar with the import phase will solve it. Decoys should cost only a diagnosis step, not rely on hidden requirements.

Primary implementation references: [M].

#### Probe imports-args-wrong

let lib = import ./lib; in
(lib.evalModules { modules = [
 { _module.args.featureModule = { options.x = lib.mkOption { default = 7; }; }; }
 ({ featureModule, ... }: { imports = [ featureModule ]; })
]; }).config.x

Observed exit 1 in 0.0179 seconds.

Stderr excerpt:

       error: infinite recursion encountered
       at /tmp/nixbench-candidate-research/lib/modules.nix:652:66:
          651|       extraArgs = mapAttrs (
          652|         name: _: addErrorContext (context name) (args.${name} or config._module.args.${name})
             |                                                                  ^
          653|       ) (functionArgs f);

#### Probe imports-args-right

let lib = import ./lib; in
(lib.evalModules {
 specialArgs.featureModule = { options.x = lib.mkOption { default = 7; }; };
 modules = [ ({ featureModule, ... }: { imports = [ featureModule ]; }) ];
}).config.x

Observed exit 0 in 0.0231 seconds.

Stdout:

7

7. module-priority-submodule-apply

Category: modules. Difficulty: hard.

A normal list definition discards mkDefault definitions; mkBefore cannot restore a list discarded by mkForce. A numerically lower mkOverride defeats mkForce. apply runs after merging, even after forced definitions. attrsOf submodule defaults populate present entries but do not create absent entries. mkIf around a scalar list element is valid in this pin; mkIf around a list used as one element leaves a nested list.

Symptom-driven prompt sketch

The generated service manifest keeps losing fallback listeners, duplicates the normalization suffix, and fails only when an optional listener is enabled. Repair services.nix so defaults, explicit operator settings, and emergency overrides behave as documented in the sample configurations. The final manifest should normalize each merged service once, include defaults only for present services, and preserve listener ordering.

Evaluator and alternate-input probes

Use real evalModules with attrsOf submodules. Probe absent versus empty instances, two service names, port and label defaults, mkDefault/ordinary/mkForce/mkOverride 40 settings, and reordered module imports. Include mkBefore/mkAfter contributions at equal priority and force plus apply. Test optional elements both enabled and disabled; a false condition alone would hide a nested-list mistake. Include two equivalent module fragments so applying normalization before merge would duplicate the suffix. A lazyAttrsOf substitution must not leave disabled phantom entries in exported instance names.

Valid alternatives

Accept whole-list mkIf, scalar-element mkIf, optional/optionals, and mkMerge where their behavior matches. Accept declarative defaults and equivalent derived-output normalization. Never reject all mkIf-inside-list syntax. Normalization may use apply or an equivalent final projection if the public option itself has the promised value.

Expected discrimination

Moderate-to-high relative potential. Memorized priority rules are insufficient when defaults and transformations cross submodule boundaries. This is the broadest candidate and needs a small, readable starter to avoid becoming a checklist task.

Primary implementation references: [M], [T].

#### Probe module-priorities

let lib = import ./lib; in
let
  eval = defs: (lib.evalModules { modules = [
    { options.x = lib.mkOption { type = lib.types.listOf lib.types.str;
        default = [ "option" ]; apply = xs: xs ++ [ "applied" ]; }; }
  ] ++ map (x: { config.x = x; }) defs; }).config.x;
in {
  defaultOnly = eval [ (lib.mkDefault [ "fallback" ]) ];
  normal = eval [ (lib.mkDefault [ "fallback" ]) [ "normal" ] ];
  force = eval [ (lib.mkBefore [ "before" ]) (lib.mkForce [ "forced" ]) ];
  lowerNumber = eval [ (lib.mkForce [ "forced" ]) (lib.mkOverride 40 [ "emergency" ]) ];
  ordered = eval [ (lib.mkAfter [ "after" ]) (lib.mkBefore [ "before" ]) ];
}

Observed exit 0 in 0.0178 seconds.

Stdout:

{"defaultOnly":["fallback","applied"],"force":["forced","applied"],"lowerNumber":["emergency","applied"],"normal":["normal","applied"],"ordered":["before","after","applied"]}

#### Probe list-nested-wrong

let lib = import ./lib; in
(lib.evalModules { modules = [
 { options.x = lib.mkOption { type = lib.types.listOf lib.types.str; }; }
 { x = [ (lib.mkIf true [ "live" ]) ]; }
]; }).config.x

Observed exit 1 in 0.0258 seconds.

Stderr excerpt:

       error: A definition for option `x."[definition 1-entry 1]"' is not of type `string'. Definition values:
       - In `<unknown-file>':
           [
             "live"
           ]

#### Probe list-condition

let lib = import ./lib; in
let
 eval = value: (lib.evalModules { modules = [
   { options.x = lib.mkOption { type = lib.types.listOf lib.types.str; }; }
   { x = value; }
 ]; }).config.x;
in { value = eval [ (lib.mkIf false "ghost") "live" ];
     whole = eval (lib.mkMerge [ (lib.mkIf false [ "ghost" ]) [ "live" ] ]); }

Observed exit 0 in 0.0238 seconds.

Stdout:

{"value":["live"],"whole":["live"]}

#### Probe submodule-list-enable

let lib = import ./lib; in
let
 eval = enabled: (lib.evalModules { modules = [
 { options.names = lib.mkOption { type = lib.types.listOf lib.types.str; default = []; }; }
 { names = [ (lib.mkIf enabled "optional") "always" ]; }
 ]; }).config.names;
in { off = eval false; on = eval true; }

Observed exit 0 in 0.0231 seconds.

Stdout:

{"off":["always"],"on":["optional","always"]}

#### Probe attrs-presence

let lib = import ./lib; in
let
 eval = type: (lib.evalModules { modules = [
 { options.instances = lib.mkOption { type = type (lib.types.submodule {
     options.port = lib.mkOption { type = lib.types.int; default = 80; };
   }); default = {}; }; }
 { instances.ghost = lib.mkIf false {}; instances.live = {}; }
 ]; }).config.instances;
in { strict = builtins.attrNames (eval lib.types.attrsOf);
     lazy = builtins.attrNames (eval lib.types.lazyAttrsOf); }

Observed exit 0 in 0.0241 seconds.

Stdout:

{"lazy":["ghost","live"],"strict":["live"]}

8. flake-nested-follows-identity

Category: flakes. Difficulty: hard.

inputs.tool.inputs.nixpkgs.follows changes an edge within the tool dependency graph. inputs.tool.follows redirects the whole tool input. Both can make marker strings equal, but the latter loses the tool outputs. Equal marker values alone are therefore a weak evaluator. The probes use actual local flakes and lock graphs, not an imitation outputs call.

Symptom-driven prompt sketch

The root and the tool report different dependency identities. An attempted lockfile cleanup makes the identities agree but removes the compiler output. Repair flake.nix so the tool shares the root dependency and still exports its compiler. The supplied local fixtures reproduce the graph without downloading nixpkgs.

Evaluator and alternate-input probes

Use three tiny local flakes, two dependency markers and one tool. Generate locked fixture graphs in evaluator-owned scratch space with nix flake lock --offline, then evaluate with --no-write-lock-file. Probe a different root marker, an additional unrelated tool output, and a root dependency update without editing the tool. Compare the resolved dependency identity and required tool outputs; inspect the lock graph's root-relative follows path if necessary, but do not require lock node names or JSON ordering. Never overwrite a candidate implementation to inject the expected follows declaration.

Valid alternatives

Accept the canonical nested follows declaration and equivalent local input indirection that preserves shared graph identity after root updates. Matching an initial URL or marker without shared resolution is insufficient. Accept regenerated lockfiles and equivalent node naming.

Expected discrimination

Moderate-to-high relative potential. The incorrect top-level follows repair looks successful under a shallow marker check. Real graph evaluation adds discrimination beyond the current fake-flake tasks, but follows syntax itself is well known.

The exact companion fixture files and reproduction commands appear in the appendix. These fixtures are stand-ins for dependencies, not a nixpkgs checkout. Root self versus lexical ./. observations are also used in the host-output candidate.

Primary implementation references: [FL].

#### Probe flake-unshared

Command:

nix eval --offline --json --no-write-lock-file path:/tmp/nixbench-candidate-research/flake-fixtures/unshared#report

This is the fixture root flake.nix; companion files are in the appendix.

{
 inputs.nixpkgs.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/dep-a";
 inputs.tool.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/tool";

 outputs = { self, nixpkgs, tool }: { report = {
   root = nixpkgs.marker; nested = tool.marker;
   identity = tool.toolIdentity or "missing";
   rootSelfMatchesDot = toString self == toString ./.;
   imported = import ./nix/location.nix { inherit self; };
 }; };
}

Observed exit 0 in 0.0272 seconds.

Stdout:

{"identity":"compiler","imported":{"localIsRoot":false,"parentIsRoot":true},"nested":"B","root":"A","rootSelfMatchesDot":true}

#### Probe flake-shared

Command:

nix eval --offline --json --no-write-lock-file path:/tmp/nixbench-candidate-research/flake-fixtures/shared#report

This is the fixture root flake.nix; companion files are in the appendix.

{
 inputs.nixpkgs.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/dep-a";
 inputs.tool.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/tool";
 inputs.tool.inputs.nixpkgs.follows = "nixpkgs";
 outputs = { self, nixpkgs, tool }: { report = {
   root = nixpkgs.marker; nested = tool.marker;
   identity = tool.toolIdentity or "missing";
   rootSelfMatchesDot = toString self == toString ./.;
   imported = import ./nix/location.nix { inherit self; };
 }; };
}

Observed exit 0 in 0.0225 seconds.

Stdout:

{"identity":"compiler","imported":{"localIsRoot":false,"parentIsRoot":true},"nested":"A","root":"A","rootSelfMatchesDot":true}

#### Probe flake-replaced

Command:

nix eval --offline --json --no-write-lock-file path:/tmp/nixbench-candidate-research/flake-fixtures/replaced#report

This is the fixture root flake.nix; companion files are in the appendix.

{
 inputs.nixpkgs.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/dep-a";

 inputs.tool.follows = "nixpkgs";
 outputs = { self, nixpkgs, tool }: { report = {
   root = nixpkgs.marker; nested = tool.marker;
   identity = tool.toolIdentity or "missing";
   rootSelfMatchesDot = toString self == toString ./.;
   imported = import ./nix/location.nix { inherit self; };
 }; };
}

Observed exit 0 in 0.0276 seconds.

Stdout:

{"identity":"missing","imported":{"localIsRoot":false,"parentIsRoot":true},"nested":"A","root":"A","rootSelfMatchesDot":true}

9. string-command-dependency-context

Category: nix-language. Difficulty: hard.

String equality compares text, not the dependency context. unsafeDiscardStringContext can make a command look unchanged while removing its build dependency. toString of a filesystem path keeps its host location without importing it; interpolation copies the path to the store and carries context. Neither blanket coercion rule is right for every field.

Symptom-driven prompt sketch

The command renderer's golden text test passes, but the generated command no longer brings its tool and configuration into the build closure. Its display-only workspace label also unexpectedly became a store path. Repair render.nix so the executable command retains its dependencies while the workspace label continues to identify the original local directory.

Evaluator and alternate-input probes

Pass actual dummy derivations and a temporary config directory. Assert command text plus builtins.getContext separately, including the derivation output and copied source reference. Use two tools with different drv identities, quote-containing filenames, and path inputs supplied at runtime. Change source bytes and require the command's referenced source identity to change. The display label must match toString of the supplied local path without forcing an unrelated file. Optionally inspect a derived consumer's inputDrvs through nix derivation show without building.

Valid alternatives

Accept interpolation, context-preserving string combinators, or explicit appendContext when the exact required dependencies remain. Accept different shell quoting with equivalent argv. Do not compare only hasContext, since an unrelated dependency would pass. Do not insist that every human-readable label have context.

Expected discrimination

Moderate-to-high relative potential. Text snapshot tests hide the defect completely, and a blanket switch between toString and interpolation breaks the other field. Agents that inspect getContext should solve it quickly.

Primary implementation references: [N].

#### Probe string-context

let
 d = derivation { name = "tool"; system = "x86_64-linux"; builder = "/not-used"; };
 command = "${d}/bin/tool";
 stripped = builtins.unsafeDiscardStringContext command;
 path = ./assets;
in {
 equalText = command == stripped;
 originalContext = builtins.hasContext command;
 strippedContext = builtins.hasContext stripped;
 localPath = builtins.toString path;
 copiedPath = "${path}";
 localContext = builtins.hasContext (builtins.toString path);
 copiedContext = builtins.hasContext "${path}";
}

Observed exit 0 in 0.0265 seconds.

Stdout:

{"copiedContext":true,"copiedPath":"/nix/store/cviqy4rpl7i8snsphq0mrj9qvwsk8gx0-assets","equalText":true,"localContext":false,"localPath":"/tmp/nixbench-candidate-research/assets","originalContext":true,"strippedContext":false}

10. overlay-composed-final-prev

Category: overlays. Difficulty: hard.

composeManyExtensions composes overlays left to right. prev contains preceding contributions; final is the completed fixed point. A consumer tied to prev.dep freezes the earlier dependency, while changing dep itself via final.dep would feed back into its own definition. // composition also replaces nested attributes instead of merging them.

Symptom-driven prompt sketch

The first overlay works by itself. After the vendor and deployment overlays are composed, the application still embeds the old dependency identity, although the package set shows the new one. Repair overlay.nix so consumers see the completed dependency and the base extension still applies exactly once. The vendor's existing nested metadata must survive.

Evaluator and alternate-input probes

Use real fix, extends, and composeManyExtensions. Compose the candidate with independent pre- and post-overlays carrying opaque dependency objects. Vary the number and order of the unrelated overlays, append another consumer override, and probe preserved metadata. Require base derivation from prev and consumers from the completed set through behavior. Add a recursion-rejection case and a shallow nested-metadata replacement case.

Valid alternatives

Accept explicit overlay folds, composeExtensions, or composeManyExtensions with equivalent order. Do not assert the source contains final or prev or count override calls. Directly recomputing a consumer is valid if it observes the final dependency and preserves the public extension behavior.

Expected discrimination

Moderate relative potential. Frontier agents know final versus prev, and the existing corpus already tests part of it. Multiple genuine composition steps and transitive consumers are needed for a meaningful increase in difficulty.

Primary implementation references: [F].

#### Probe overlays

let lib = import ./lib; in
let
  base = self: { dep = "A"; consumer = "uses-${self.dep}"; };
  first = final: prev: { dep = prev.dep + "B";
    stale = prev.dep; live = final.dep; };
  last = final: prev: { dep = prev.dep + "C"; };
  p = lib.fix (lib.extends (lib.composeManyExtensions [ first last ]) base);
in { inherit (p) dep consumer stale live; }

Observed exit 0 in 0.0194 seconds.

Stdout:

{"consumer":"uses-ABC","dep":"ABC","live":"ABC","stale":"A"}

11. source-filter-traversal-stability

Category: flakes. Difficulty: hard.

builtins.path does not descend into excluded directories. A filter selecting only .nix filenames therefore produces a valid-looking but empty source. Allowing all directories fixes traversal but preserves empty excluded directories. lib.fileset can express a narrower source; compare its coerced store path, not the entire source attrset that also records its origin.

Symptom-driven prompt sketch

The source archive evaluates successfully, but src/main.nix is missing. Another attempt includes the source but changes its store identity whenever a documentation file changes. Repair source.nix so it exports the runtime source tree, preserves its relative layout, and ignores documentation-only edits. The demo has two copies of the same source with different scratch-directory names.

Evaluator and alternate-input probes

Create three fixture trees with identical basenames for the selected store source name: two differ only in ignored contents, and the third changes an included file. Check relative file contents and a semantic directory listing, not only existence of the source store path. Include nested source directories, an empty selected tree, a non-Nix asset under the runtime source subtree, and an ignored directory whose name ends in .nix. Compare toString of the returned source for identity. Require a stable explicit source name if relocation invariance is promised.

Valid alternatives

Accept builtins.path with a correct traversal-aware filter, lib.fileset.toSource, or lib.cleanSourceWith with equivalent selection and stable naming. Unless the prompt requires omitting empty directories, do not reject harmless empty ignored directories. Do not demand a specific NAR hash or compare entire fileset source attrsets.

Expected discrimination

Moderate relative potential. The missing-directory bug is familiar, but the relocation and ignored-edit invariants catch superficial repairs. Keep exact source selection public; hidden exclusion rules would make the task unfair.

Fixture setup is in the appendix. The probe shows both the directory-traversal repair and the stricter fileset result; the final prompt must choose whether empty ignored directories are observable.

Primary implementation references: [N], [FS].

#### Probe filtered-path

let lib = import ./lib; in
let
 bad = builtins.path { path = ./tree-a; name = "source";
   filter = path: type: lib.hasSuffix ".nix" (toString path); };
 good = path: builtins.path { inherit path; name = "source";
   filter = path: type: type == "directory" || lib.hasSuffix ".nix" (toString path); };
 fs = path: lib.fileset.toSource { root = path; fileset = path + "/src"; };
in {
 badHasMain = builtins.pathExists (bad + "/src/main.nix");
 goodHasMain = builtins.pathExists (good ./tree-a + "/src/main.nix");
 directoriesRemain = builtins.pathExists (good ./tree-a + "/docs");
 filesetDropsDocs = !(builtins.pathExists (fs ./tree-a + "/docs"));
 ignoredStable = good ./tree-a == good ./tree-b;
 includedChanges = good ./tree-a != good ./tree-c;
 filesetStable = toString (fs ./tree-a) == toString (fs ./tree-b);
}

Observed exit 0 in 0.0323 seconds.

Stdout:

{"badHasMain":false,"directoriesRemain":true,"filesetDropsDocs":true,"filesetStable":true,"goodHasMain":true,"ignoredStable":true,"includedChanges":true}

12. module-migration-assertion-gate

Category: modules. Difficulty: hard.

A rename is not ordinary assignment from a final config value. The real rename module forwards definitions and their priority, so a default on the old name yields to an explicit new value, while conflicting ordinary values remain a conflict. mkRemovedOptionModule records a failing assertion. Bare evalModules does not make that assertion fatal, and it does not supply NixOS assertions/warnings option declarations.

Symptom-driven prompt sketch

The compatibility layer silently overwrites a user's new setting when an older module is imported. It also accepts a removed setting that the release notes say must fail. Repair compat.nix and evaluate.nix so old callers retain normal option precedence, contradictory explicit settings produce a useful failure, and removed settings stop the exported configuration.

Evaluator and alternate-input probes

Declare warnings and assertions in the fixture, use real rename/remove helpers, and force lib.asserts.checkAssertWarn at the public export boundary. Probe old-only, new-only, old mkDefault plus new ordinary, old ordinary plus new mkForce, conflicting ordinary values, and removed option absent/present. Require that unrelated assertions from an injected module are also enforced. Inspect warning content for the old/new option paths without depending on full wording. A successful config projection before forcing assertions is a negative fixture, not a passing result.

Valid alternatives

Accept native helpers or equivalent definition-preserving forwarding and explicit assertion enforcement. Accept any actionable message that names the public problematic setting. Reject deleting assertions, swallowing all errors, or promoting renamed defaults to ordinary definitions. Do not require the full NixOS module tree.

Expected discrimination

Moderate-to-high relative potential. Direct config copying and assuming evalModules enforces NixOS assertions are both plausible. Agents that understand the separation between the generic module system and NixOS activation should pass.

Primary implementation references: [M], [A].

#### Probe migration

let lib = import ./lib; in
let
 schema = { options.new = lib.mkOption { type = lib.types.int; default = 0; };
   options.warnings = lib.mkOption { type = lib.types.listOf lib.types.str; default = []; }; };
 rename = lib.mkRenamedOptionModule [ "old" ] [ "new" ];
 eval = defs: (lib.evalModules { modules = [ schema rename ] ++ defs; }).config;
in { old = (eval [ { old = 7; } ]).new;
     priority = (eval [ { old = lib.mkDefault 7; } { new = 8; } ]).new; }

Observed exit 0 in 0.0208 seconds.

Stdout:

{"old":7,"priority":8}

#### Probe migration-conflict

let lib = import ./lib; in
(lib.evalModules { modules = [
 { options.new = lib.mkOption { type = lib.types.int; default = 0; };
   options.warnings = lib.mkOption { type = lib.types.listOf lib.types.str; default = []; }; }
 (lib.mkRenamedOptionModule [ "old" ] [ "new" ])
 { old = 7; new = 8; }
]; }).config.new

Observed exit 1 in 0.0259 seconds.

Stderr excerpt:

       error: The option `new' has conflicting definition values:
       - In `<unknown-file>': 8
       - In `<unknown-file>': 7
       Use `lib.mkForce value` or `lib.mkDefault value` to change the priority on any of these definitions.

#### Probe removed-assertions

let lib = import ./lib; in
let
 r = lib.evalModules { modules = [
 { options.assertions = lib.mkOption { type = lib.types.listOf lib.types.unspecified; default = []; }; }
 (lib.mkRemovedOptionModule [ "obsolete" ] "Use new instead.")
 { obsolete = true; }
 ]; };
in { evaluated = true; checks = map (x: x.assertion) r.config.assertions;
     gate = builtins.tryEval (assert builtins.all (x: x.assertion) r.config.assertions; "ok"); }

Observed exit 0 in 0.0215 seconds.

Stdout:

{"checks":[false],"evaluated":true,"gate":{"success":false,"value":false}}

13. lazy-recursive-update-closure

Category: nix-language. Difficulty: hard.

A rec attrset computes label against its own original version. Updating version with // leaves label bound to the original recursive scope. Moving the same closure into a recursive let does not automatically make later updates rebind it. Within rec, version = version shadows an outer binding and loops; inherit version does not. Nested // also discards sibling fields.

Symptom-driven prompt sketch

After changing the release version, the manifest says version 2 but its label still says v1. The next attempted fix causes infinite recursion, and another drops the metadata owner. Repair release.nix so each supplied update recomputes dependent fields from the completed record and retains unrelated nested metadata. The caller can apply several updates in sequence.

Evaluator and alternate-input probes

Probe two successive version updates, a caller-supplied explicit label, nested metadata updates with unknown siblings, and an unused throwing metadata field. The public contract must distinguish derived labels from explicitly overridden labels. Include renamed outer variables to catch accidental lexical capture. Compare outputs across update order where independent updates should commute, and force only fields promised by the interface.

Valid alternatives

Accept a constructor reapplied to updated inputs, a proper fixed point, or explicit recomputation of all documented derived fields. Accept ordinary let bindings when they implement the same update semantics. Do not require rec or lib.fix. Avoid recursiveUpdate if it forces or rewrites values beyond the contract.

Expected discrimination

Moderate relative potential. The minimal example is easy; repeated updates plus explicit overrides make it discriminating. This should not be activated as another one-line infinite-recursion repair.

Primary implementation references: [N], [F].

#### Probe lazy-cycle

let
  original = rec { version = "1"; label = "v${version}"; };
  mk = self: { version = "1"; label = "v${self.version}"; };
  fix = f: let x = f x; in x;
in { shallow = (original // { version = "2"; }).label;
     rebound = (fix (self: mk self // { version = "2"; })).label;
     nested = { meta = { a = 1; b = 2; }; } // { meta = { b = 3; }; };
}

Observed exit 0 in 0.0136 seconds.

Stdout:

{"nested":{"meta":{"b":3}},"rebound":"v2","shallow":"v1"}

#### Probe rec-shadow

let version = "2"; in (rec { version = version; }).version

Observed exit 1 in 0.0164 seconds.

Stderr excerpt:

       error: infinite recursion encountered
       at /tmp/nixbench-candidate-research/rec-shadow.nix:1:40:
            1| let version = "2"; in (rec { version = version; }).version
             |                                        ^

#### Probe lexical-right

let lib = import ./lib; in
let
 version = "2";
 mk = self: { version = "1"; label = "v${self.version}"; meta = { keep = 1; changed = 2; }; };
 patched = lib.fix (self: let base = mk self; in base // {
   version = "2"; meta = base.meta // { changed = 3; };
 });
in { plain = { version = version; }; recursiveInherit = rec { inherit version; };
     inherit (patched) label meta; }

Observed exit 0 in 0.0199 seconds.

Stdout:

{"label":"v2","meta":{"changed":3,"keep":1},"plain":{"version":"2"},"recursiveInherit":{"version":"2"}}

14. debug-contextful-attribute-name

Category: debugging. Difficulty: hard.

The visible error says a string is not allowed to refer to a store path. The failing operation is creating an attribute name, not copying a path or launching a build. Using a semantic context-free identifier fixes that key; stripping context from the whole rendered record would hide the key error while losing command dependencies.

Symptom-driven prompt sketch

The generated command registry fails inside listToAttrs with the string ... is not allowed to refer to a store path. The trace points to the registry entry name. Nearby files contain an absolute builder path and sandbox settings, but no build has started. Repair registry.nix so callers can select commands by their declared IDs and each command still carries its tool dependency.

Evaluator and alternate-input probes

Supply real dummy derivations with separate public IDs, duplicate package names under distinct IDs, and opaque passthru metadata. Verify registry keys against the supplied IDs, command text, and exact getContext dependency entries. Rename a tool without changing its ID and require key stability. Change a tool derivation while preserving its display name and require the command context to change. Add wrong fixtures for globally discarding context and keying by outPath.

Valid alternatives

Accept the supplied context-free ID or equivalent declared naming scheme. Context removal restricted to a key can be valid only if it preserves the public ID contract; do not ban unsafeDiscardStringContext by token. Accept any context-preserving command construction.

Expected discrimination

Moderate relative potential. The store-path wording encourages irrelevant purity or sandbox edits. The most useful discriminator is retaining context in values after fixing the name. This overlaps the command-context task; calibrate both and keep only one if outcomes are redundant.

Primary implementation references: [N].

#### Probe context-name-wrong

let d = derivation { name = "tool"; system = "x86_64-linux"; builder = "/not-used"; };
in builtins.listToAttrs [ { name = "${d}"; value = 1; } ]

Observed exit 1 in 0.0240 seconds.

Stderr excerpt:

       error: the string '/nix/store/bd17nj5x4hg0hdpas0wcvz4kik80r7lc-tool' is not allowed to refer to a store path (such as '/nix/store/q207bjzgh0g5387s74pcyl0xlfplq34k-tool.drv^out')

#### Probe context-name-right

let d = derivation { name = "tool"; system = "x86_64-linux"; builder = "/not-used"; };
in { ${d.name} = builtins.hasContext "${d}/bin/tool"; }

Observed exit 0 in 0.0259 seconds.

Stdout:

{"tool":true}

15. fetcher-fixed-output-identity

Category: purity. Difficulty: hard.

Fixed-output outPath identity depends on the declared content hash, mode, and output name, not merely the builder recipe or URL. Changing a URL can change drvPath while leaving outPath unchanged. Flat byte hashing and recursive NAR hashing are different contracts. These eval probes prove identity and stale expected bytes; they do not execute a fetch or reproduce a build-time mismatch.

Symptom-driven prompt sketch

A release-source update changed the URL, but the source path stayed the same on a machine with a populated store. Another machine reported a hash mismatch. Repair source-spec.nix using the provided local release bytes and the documented raw-file format so the expected content, hash mode, and source identity agree for each release. No download is needed for the reproducer.

Evaluator and alternate-input probes

Supply two local byte fixtures and independent expected SHA-256 values computed by the hidden evaluator. Read the candidate's source descriptor or derivation attrs without realizing it. Probe URL-only changes, content changes under the same URL, a renamed output, flat versus recursive modes, and hex versus equivalent SRI hashes. Require a content change to update the expected hash and output identity, while mirror-only changes preserve content identity. For unpacking fetchers, use precomputed NAR fixtures and a documented unpacked-tree contract, not hashFile on the archive.

Valid alternatives

Accept valid SRI or hex encodings after normalization, equivalent fetcher wrappers with the specified semantics, and any pure descriptor constructor. Do not require a fake hash or a real failing build. Do not label a fabricated throw as Nix's hash-mismatch error.

Expected discrimination

Moderate relative potential. This tests reasoning from conflicting cache evidence rather than memorizing a fetcher schema. The evaluator can establish declared content consistency, but cannot prove builder behavior or actual download integrity with eval alone. Keep that limitation in the eventual task description.

Primary implementation references: [N], [DER].

#### Probe fixed-output

let
 hash = builtins.hashString "sha256" "old-content";
 mk = url: outputHash: outputHashMode: derivation {
   name = "source"; system = "x86_64-linux"; builder = "/not-used";
   inherit url outputHash outputHashMode; outputHashAlgo = "sha256";
 };
 old = mk "https://example.invalid/v1" hash "flat";
 changed = mk "https://example.invalid/v2" hash "flat";
 right = mk "https://example.invalid/v2" (builtins.hashString "sha256" "new-content") "flat";
in {
 staleOutputSame = old.outPath == changed.outPath;
 recipeDifferent = old.drvPath != changed.drvPath;
 correctedOutputDifferent = right.outPath != old.outPath;
 flatVsRecursiveDifferent = old.outPath != (mk "https://example.invalid/v1" hash "recursive").outPath;
 staleMatchesNewBytes = hash == builtins.hashString "sha256" "new-content";
}

Observed exit 0 in 0.0220 seconds.

Stdout:

{"correctedOutputDifferent":true,"flatVsRecursiveDifferent":true,"recipeDifferent":true,"staleMatchesNewBytes":false,"staleOutputSame":true}

16. lazy-type-error-boundary

Category: nix-language. Difficulty: hard.

Wrapping integer-plus-string in tryEval still terminates evaluation. The same wrapper catches throw and returns success = false. A validator must check supported input shapes before type-sensitive operations; adding deepSeq only changes strictness, not which evaluator exceptions are catchable.

Symptom-driven prompt sketch

The batch validator handles explicit rejected values but aborts the entire batch when one record contains a string where a count should be. The code already uses tryEval. Repair validate.nix so every supported malformed data shape returns the documented per-record rejection while later records still produce results. Disabled records may contain unavailable data and must remain unforced.

Evaluator and alternate-input probes

Probe integers, strings, lists, null, missing attrs, and nested malformed fields using the public input grammar. Add catchable throws only where the public API supports deferred values. Test valid records after malformed records and a disabled poison record. Check that a thrown exported value is rejected, but an unrelated field is never evaluated. Execute fatal candidate evaluations in subprocesses; an evaluator's own tryEval wrapper cannot catch all candidate failures.

Valid alternatives

Accept type guards, a structural validator, or parsing into a tagged data representation before arithmetic. Do not demand a universal catch function for arbitrary Nix expressions; that contract is impossible with tryEval. A candidate may reject unsupported raw functions if the prompt defines a data-only API.

Expected discrimination

Moderate relative potential. Many agents overgeneralize try/catch, but shape validation is straightforward once the failing primitive is isolated. Preserve this as a candidate only if the batch and laziness probes expose more than a single missing isInt guard.

Primary implementation references: [N].

#### Probe uncatchable-type

builtins.tryEval (1 + "x")

Observed exit 1 in 0.0183 seconds.

Stderr excerpt:

       error: cannot add a string to an integer
       at /tmp/nixbench-candidate-research/uncatchable-type.nix:1:23:
            1| builtins.tryEval (1 + "x")
             |                       ^

#### Probe catchable-throw

builtins.tryEval (throw "x")

Observed exit 0 in 0.0191 seconds.

Stdout:

{"success":false,"value":false}

#### Probe guarded-value

let
 addOne = x: if builtins.isInt x then { valid = true; value = x + 1; }
             else { valid = false; value = null; };
in { good = addOne 1; bad = addOne "x"; }

Observed exit 0 in 0.0202 seconds.

Stdout:

{"bad":{"valid":false,"value":null},"good":{"valid":true,"value":2}}

17. string-shell-template-roundtrip

Category: nix-language. Difficulty: hard.

Wrapping data in single quotes fails when the data contains a quote. Indented strings strip the minimum indentation among content lines; a column-zero heredoc terminator changes the indentation of the whole generated script. Nix escaping of a literal shell ${...} does not perform shell argument quoting. The probes show lib.escapeShellArg and toShellVars preserving quotes, dollar signs, newlines, empty values, and array elements.

Symptom-driven prompt sketch

The generated startup script works for the sample title but fails on an apostrophe. Its heredoc payload gained leading spaces after a formatting change, and ${TITLE} must still be expanded by the shell at runtime. Repair script.nix so generated declarations preserve supplied values exactly and the generated payload has the documented bytes. The renderer emits only the documented assignment grammar: scalar assignments and indexed-array declarations with literal quoted values, without executing substitutions or other shell statements.

Evaluator and alternate-input probes

Keep the hidden check entirely in nix eval. Supply values containing quotes, dollar signs, newlines, empty strings, and spaces in array elements. Decode the candidate declarations with an independent finite-state Nix parser for the public literal-assignment grammar, then compare the decoded scalars and arrays to the original values. Support concatenated single-quoted, double-quoted, and backslash-escaped literal segments; declare the supported grammar in the prompt. Reject expansions and executable statements as outside that data format. Compare the heredoc body and literal runtime interpolation bytes separately. The parser itself needs contract fixtures covering equivalent quote spellings before this task is ready; it is not implemented by this research report.

Valid alternatives

Accept escapeShellArg, toShellVars, or equivalent literal quoting within the documented assignment grammar. Accept different indentation and heredoc strategies with the same payload and runtime interpolation. Do not require library helper tokens or an exact quote spelling. General executable shell expressions, including command substitution, are outside the public renderer format; without that restriction, a Nix-only decoder could not accept every semantically equivalent shell program.

Expected discrimination

Moderate relative potential. Each issue is common, but fixing only the Nix escape often leaves shell quoting wrong. This is a weaker discriminator if the task simply tells the agent to call toShellVars. The independent literal decoder adds evaluator work and is a readiness risk; the task should remain a reserve until those parser contracts exist.

I also evaluated the emitted declarations in Bash. Exit code was 0, and the NUL-delimited output was b"a'b $HOME\nnext\x00\x00a b\x00c'd\x00". This corroborates quoting; the core Nix outputs below are the requested eval evidence. The optional Bash check stays offline and does not build derivations.

Primary implementation references: [N], [S].

#### Probe shell-wrong

let lib = import ./lib; in
let value = "a'b $HOME";
in { wrong = "'${value}'"; right = lib.escapeShellArg value; }

Observed exit 0 in 0.0202 seconds.

Stdout:

{"right":"'a'\\''b $HOME'","wrong":"'a'b $HOME'"}

#### Probe indent-wrong

{
 wrong = ''
      cat <<EOF
      payload
EOF
    '';
 right = ''
      cat <<EOF
      payload
      EOF
    '';
}

Observed exit 0 in 0.0170 seconds.

Stdout:

{"right":"cat <<EOF\npayload\nEOF\n","wrong":"      cat <<EOF\n      payload\nEOF\n"}

#### Probe shell-escaping

let lib = import ./lib; in
let
 text = "a'b $HOME\nnext";
 vars = lib.strings.toShellVars { TITLE = text; EMPTY = ""; ARGS = [ "a b" "c'd" ]; };
 script = ''
     printf '%s\n' "''${TITLE}"
       printf '%s\n' "''${EMPTY}"
   '';
in { inherit vars script; quoted = lib.escapeShellArg text; }

Observed exit 0 in 0.0215 seconds.

Stdout:

{"quoted":"'a'\\''b $HOME\nnext'","script":"printf '%s\\n' \"${TITLE}\"\n  printf '%s\\n' \"${EMPTY}\"\n","vars":"declare -a ARGS=('a b' 'c'\\''d')\nEMPTY=''\nTITLE='a'\\''b $HOME\nnext'"}

18. flake-host-system-output-boundary

Category: flakes. Difficulty: hard.

A helper mapping systems over all outputs nests nixosConfigurations under system keys and can pass that loop system into every host. Host outputs are keyed by host name; each host must receive its declared target system. A path literal inside an imported subdirectory refers to that directory, while self refers to the flake root. At the root flake.nix, self and ./. can coincide, so replacing every ./. is not a valid rule.

Symptom-driven prompt sketch

The board configuration disappeared from the expected flake output path after a per-system refactor. A debug projection also says the ARM board was constructed with an x86 system, and its assets resolve under nix/ instead of the repository root. Repair outputs.nix so package outputs follow requested systems, hosts keep their declared systems, and imported helpers receive the intended source root.

Evaluator and alternate-input probes

Call the candidate outputs constructor with multiple system lists and an independent host-to-system map, then run the local flake entry point for output-path semantics. Inject a recording nixosSystem stand-in that asserts the supplied system agrees with the host contract and records the passed module source; do not pretend this is full NixOS evaluation. Include a host whose target is absent from the package system list, changed host names, and a helper moved one directory deeper. Verify source file contents through the passed root and reject accidental system nesting.

Valid alternatives

Accept separate package and host maps, a combined outputs helper that treats system-independent outputs correctly, or an explicit builder per host. Accept self, a passed root path, or a correctly rooted relative path. Do not require self textually or assume every nixosConfigurations entry must be built for the machine evaluating it.

Expected discrimination

Moderate relative potential. Existing tasks already cover per-system outputs and host system selection, so only the independent host map plus lexical source-root evidence adds much. This tests wiring consistency through a stand-in; a real NixOS hostPlatform validation would require more vendored modules and is out of the proposed evaluator scope.

Primary implementation references: [FL], [N].

#### Probe host-system

let lib = import ./lib; in
let
 systems = [ "x86_64-linux" "aarch64-linux" ];
 hosts = { laptop = "x86_64-linux"; board = "aarch64-linux"; };
 mkHost = system: target: (lib.evalModules { modules = [
   { options.system = lib.mkOption { type = lib.types.str; };
     options.target = lib.mkOption { type = lib.types.str; }; }
   { inherit system target; }
 ]; }).config;
 wrong = lib.genAttrs systems (system: {
   nixosConfigurations = lib.mapAttrs (_: target: mkHost system target) hosts;
 });
 right = lib.mapAttrs (_: system: mkHost system system) hosts;
in {
 wrongRoots = builtins.attrNames wrong;
 wrongBoard = wrong.x86_64-linux.nixosConfigurations.board;
 rightBoard = right.board;
}

Observed exit 0 in 0.0219 seconds.

Stdout:

{"rightBoard":{"system":"aarch64-linux","target":"aarch64-linux"},"wrongBoard":{"system":"x86_64-linux","target":"aarch64-linux"},"wrongRoots":["aarch64-linux","x86_64-linux"]}

#### Probe flake-shared

Command:

nix eval --offline --json --no-write-lock-file path:/tmp/nixbench-candidate-research/flake-fixtures/shared#report

This is the fixture root flake.nix; companion files are in the appendix.

{
 inputs.nixpkgs.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/dep-a";
 inputs.tool.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/tool";
 inputs.tool.inputs.nixpkgs.follows = "nixpkgs";
 outputs = { self, nixpkgs, tool }: { report = {
   root = nixpkgs.marker; nested = tool.marker;
   identity = tool.toolIdentity or "missing";
   rootSelfMatchesDot = toString self == toString ./.;
   imported = import ./nix/location.nix { inherit self; };
 }; };
}

Observed exit 0 in 0.0225 seconds.

Stdout:

{"identity":"compiler","imported":{"localIsRoot":false,"parentIsRoot":true},"nested":"A","root":"A","rootSelfMatchesDot":true}

19. purity-explicit-release-inputs

Category: purity. Difficulty: medium.

In pure evaluation, builtins.currentTime is absent and getEnv returns an empty string even when the process environment contains a value. A fallback to the current clock or environment therefore either fails or silently changes meaning. Deterministic release metadata must be supplied explicitly or derived from pinned inputs.

Symptom-driven prompt sketch

The release manifest contains the intended revision locally but an empty revision in CI, where the timestamp expression also fails. Repair manifest.nix so the same declared release inputs produce the same manifest under both evaluation modes, while two different releases still receive their own revision and epoch.

Evaluator and alternate-input probes

Import the candidate from a trusted copied store path under pure evaluation. Run explicit release inputs in fresh processes with conflicting environment values, then compare repeated outputs. Vary both revision and epoch so hard-coded replacements fail. Test missing optional metadata according to a public deterministic fallback rule; do not use the real clock as an oracle or sleep between runs. The candidate's input path must itself be admitted to pure evaluation, or the test would fail before reaching its code.

Valid alternatives

Accept explicit arguments, a checked-in generated metadata file, or pinned flake metadata where the documented missing-value behavior matches. Do not ban getEnv textually if it is dead code, and do not accept a constant epoch when the API promises to use the supplied epoch.

Expected discrimination

Low-to-moderate relative potential. The purity rule is familiar and frontier agents will often solve it. The explicit-input perturbations make it sound, but it is a reserve candidate rather than a likely answer to corpus saturation.

Both environment probes ran with NIXBENCH_RELEASE=host-secret. This was a synthetic test value, not a real secret. Pure probes use --option pure-eval true --expr; the ordinary --file probe reads the process environment.

Primary implementation references: [N].

#### Probe pure-observation

Run in explicit pure mode as described above.

{ clockExists = builtins ? currentTime; env = builtins.getEnv "NIXBENCH_RELEASE"; }

Observed exit 0 in 0.0197 seconds.

Stdout:

{"clockExists":false,"env":""}

#### Probe pure-clock

Run in explicit pure mode as described above.

builtins.currentTime

Observed exit 1 in 0.0193 seconds.

Stderr excerpt:

error: attribute 'currentTime' missing
       at «string»:1:1:
            1| builtins.currentTime
             | ^

#### Probe pure-explicit

Run in explicit pure mode as described above.

let make = { revision, epoch }: { inherit revision epoch; }; in make { revision = "abc123"; epoch = 1700000000; }

Observed exit 0 in 0.0199 seconds.

Stdout:

{"epoch":1700000000,"revision":"abc123"}

#### Probe impure-environment

builtins.getEnv "NIXBENCH_RELEASE"

Observed exit 0 in 0.0152 seconds.

Stdout:

"host-secret"

20. string-literal-replacement-order

Category: nix-language. Difficulty: medium.

Replacement text is not rescanned by the same replaceStrings call. A fold of separate replacements cascades instead. When patterns overlap at the same location, input order matters. A renderer must choose the contract intentionally rather than treating the two implementations as interchangeable.

Symptom-driven prompt sketch

The template renderer changes a user-supplied literal token into another variable's value. It also partially replaces the longer of two overlapping placeholders. Repair render.nix so each occurrence in the original template is replaced once, inserted values remain literal, and longer matching placeholders take precedence. Unknown tokens remain unchanged.

Evaluator and alternate-input probes

Probe a replacement value containing another known token, overlapping token names, swapped mapping order, empty replacement values, repeated occurrences, unknown tokens, and non-ASCII literal text. Rename all tokens and values for a second fixture. Specify nonempty placeholder keys in the public grammar, since empty-pattern behavior is a separate contract. Compare final strings, including string contexts if replacements are promised to carry dependencies.

Valid alternatives

Accept a single replaceStrings call with correctly ordered patterns, a tokenizer, or another non-cascading literal renderer. Do not require a particular sorting algorithm. If placeholder delimiters make overlaps impossible, omit that requirement instead of inventing hidden cases outside the grammar.

Expected discrimination

Low-to-moderate relative potential. The trap is real but narrow, and a frontier agent with one failing example can fix it quickly. Best used as part of a larger templating repair, not given a hard label on its own.

Primary implementation references: [N].

#### Probe replace-order

let lib = import ./lib; in
let
 source = "@a@ @b@";
 from = [ "@a@" "@b@" ]; to = [ "@b@" "done" ];
in {
 once = builtins.replaceStrings from to source;
 cascade = lib.foldl' (s: pair: builtins.replaceStrings [ pair.from ] [ pair.to ] s)
   source [ { from = "@a@"; to = "@b@"; } { from = "@b@"; to = "done"; } ];
 shortFirst = builtins.replaceStrings [ "a" "ab" ] [ "X" "Y" ] "ab";
 longFirst = builtins.replaceStrings [ "ab" "a" ] [ "Y" "X" ] "ab";
}

Observed exit 0 in 0.0201 seconds.

Stdout:

{"cascade":"done done","longFirst":"Y","once":"@b@ done","shortFirst":"Xb"}

21. purity-sandbox-phase-confusion

Category: purity. Difficulty: medium.

The failure occurs while derivationStrict evaluates an attribute containing readFile, before any builder runs. __noChroot controls a build setting and cannot override pure evaluator path restrictions. The repaired expression supplies the data as an explicit input and still forces drvPath successfully.

Symptom-driven prompt sketch

The trace says an absolute machine-specific file is forbidden in pure evaluation. The starter already sets __noChroot, and the incident notes suggest disabling sandboxing. Repair config-package.nix so the caller-provided configuration determines the package without reading the developer machine. The normal derivation must still evaluate in pure mode.

Evaluator and alternate-input probes

Pass two distinct config values or admitted store paths and force the resulting drvPath plus observable config attributes under pure evaluation. Put a different synthetic value at the forbidden host path during the impure comparison; outputs must follow the explicit input, not that path. Vary the temporary directory and run twice. Do not treat merely removing __noChroot as a sufficient repair. Never change daemon sandbox configuration or perform a build.

Valid alternatives

Accept explicit data, a source path admitted to pure evaluation, or a deterministic packaged config source. A candidate can retain an irrelevant __noChroot field unless the public contract explicitly forbids it; its presence alone says nothing about evaluation purity. Reject switching the evaluator to --impure as a solution.

Expected discrimination

Low relative potential. The trace clearly identifies the phase, and the existing purity task covers similar ground. Keep as a reserve or merge its decoy into a stronger debugging task.

Primary implementation references: [N], [DER].

#### Probe nochroot-read

Run in explicit pure mode as described above.

(derivation { name = "example"; system = "x86_64-linux"; builder = "/not-used"; __noChroot = true; machine = builtins.readFile /tmp/nixbench-candidate-research/machine-file; }).drvPath

Observed exit 1 in 0.0189 seconds.

Stderr excerpt:

       error: access to absolute path '/tmp/nixbench-candidate-research/machine-file' is forbidden in pure evaluation mode (use '--impure' to override)

#### Probe nochroot-explicit-forced

Run in explicit pure mode as described above.

let d = derivation { name = "example"; system = "x86_64-linux"; builder = "/not-used"; machine = "explicit-data"; }; in { evaluated = builtins.isString d.drvPath; value = d.machine; }

Observed exit 0 in 0.0209 seconds.

Stdout:

{"evaluated":true,"value":"explicit-data"}

Reproduction fixtures

All fixture writes were confined to /tmp/nixbench-candidate-research. Only this Markdown report is intended as a repository change.

Copied source and path fixtures

Prepare lib/ from the pinned revision above. For the context probe, create assets/config containing exactly payload\n. For the filtered-path probe, create the following files. Each displayed \n is one newline byte.

For the pure-path rejection probe, machine-file contains host-only; pure evaluation rejects reading it before its contents matter. This is a synthetic fixture, not a machine credential or environment file.

Local flake fixtures

The dependency fixtures are actual flakes with deliberately small outputs. The input named nixpkgs is only a marker flake for testing the flake graph; it does not emulate nixpkgs package semantics. Absolute scratch paths keep the recorded experiment reproducible. An evaluator should generate its own local fixture paths and lockfiles offline, or use equivalent portable locked relative inputs supported by the pinned Nix version.

flake-fixtures/dep-a/flake.nix:

{ outputs = { self }: { marker = "A"; }; }

flake-fixtures/dep-b/flake.nix:

{ outputs = { self }: { marker = "B"; }; }

flake-fixtures/tool/flake.nix:

{
 inputs.nixpkgs.url = "path:/tmp/nixbench-candidate-research/flake-fixtures/dep-b";
 outputs = { self, nixpkgs }: { marker = nixpkgs.marker; toolIdentity = "compiler"; };
}

flake-fixtures/unshared/nix/location.nix:

{ self }: {
 localIsRoot = toString ./. == toString self;
 parentIsRoot = toString ../. == toString self;
}

flake-fixtures/shared/nix/location.nix:

{ self }: {
 localIsRoot = toString ./. == toString self;
 parentIsRoot = toString ../. == toString self;
}

flake-fixtures/replaced/nix/location.nix:

{ self }: {
 localIsRoot = toString ./. == toString self;
 parentIsRoot = toString ../. == toString self;
}

For each root fixture (unshared, shared, replaced), run:

nix flake lock --offline path:/tmp/nixbench-candidate-research/flake-fixtures/shared
nix eval --offline --json --no-write-lock-file \
  path:/tmp/nixbench-candidate-research/flake-fixtures/shared#report

Use path: explicitly. On this machine the bare /tmp/... reference was incorrectly discovered as a Git input rooted at /tmp; path: avoids that environmental distraction. The lock graph was generated entirely from the local fixture flakes. The benchmark evaluator must not discover or repair the candidate's graph by downloading inputs.

Findings that rule out misleading tasks

• A blanket claim that mkIf cannot appear inside a list is false for the tested library. Scalar-element conditions work and false elements disappear. The verified type error is a list-valued element under a listOf str option.
• evalModules does not automatically provide or enforce the NixOS assertions and warnings interface. The evaluator must declare and force that boundary explicitly.
• lib.makeOverridable supplies function-argument overrides and forwards an existing overrideAttrs method; it is not the stdenv fixed-point implementation. extendMkDerivation delegates to a supplied constructor and does not remove this dependency.
• self and ./. are equal as source roots in the tested root flake. The verified difference occurs in an imported subdirectory. A task should diagnose lexical location, not enforce a universal preference for self.
• fileset.toSource returns source metadata as well as a coerced path. Comparing the full attrset can report a difference between equivalent source trees because their origins differ. Compare the exported store path and contents when testing reproducibility.
• No nix eval probe can prove that a fetched source actually builds or reproduce a real download hash mismatch without realization. The fixed-output candidate checks declared content and path identity and states that limit.
• __noChroot says nothing about whether a host path can be read during pure evaluation. Changing sandbox settings would test the wrong phase.
• tryEval cannot make arbitrary candidate evaluation safe. Fatal type errors and recursion need subprocess handling in the hidden evaluator.

Primary sources

The transcripts are direct observations from the pinned tools. The following source files define the behavior being tested. Pin these references with the proposed fixture; do not silently upgrade the library while keeping expected behavior from this report.

[N]: https://github.com/NixOS/nix/blob/2.34.8/src/libexpr/primops.cc [F]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/lib/fixed-points.nix [C]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/lib/customisation.nix [D]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/pkgs/stdenv/generic/make-derivation.nix [M]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/lib/modules.nix [T]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/lib/types.nix [A]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/lib/asserts.nix [S]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/lib/strings.nix [FS]: https://github.com/NixOS/nixpkgs/blob/d2ed99647a4b195f0bcc440f76edfa10aeb3b743/lib/fileset/default.nix [FL]: https://nix.dev/manual/nix/2.34/command-ref/new-cli/nix3-flake.html [DER]: https://nix.dev/manual/nix/2.34/language/advanced-attributes.html

Useful source locations in the inspected nixpkgs checkout:

• lib/modules.nix:275 merges freeform values separately; :652 resolves module arguments; :1043 through :1075 merge definitions before apply; :1486 implements removed options; :1535 implements renames.
• lib/types.nix:754 documents lazyAttrsOf conditional-presence behavior; :804 implements attrsOf; :1088 implements deferredModule; :1192 onward constructs submodule evaluations and supplies names.
• lib/customisation.nix:154 implements makeOverridable; :604 implements makeScope; :816 implements extendMkDerivation.
• pkgs/stdenv/generic/make-derivation.nix:78 through :136 implement finalAttrs and override composition, including the warning about finalAttrs-dependent attribute names.
• lib/asserts.nix:192 implements checkAssertWarn. NixOS calls it explicitly in nixos/modules/system/activation/top-level.nix:79; a generic evalModules fixture must provide its own equivalent boundary.

The Nix source/manual links identify the owning implementation and documentation; the native builtin claims here are established by the local transcripts. No online documentation availability is required to run these proposed evaluators.

Verification accounting: 51 distinct included probes; 37 exited 0 and 14 demonstrated expected evaluation rejection. Included probe subprocess durations ranged from 0.0136 to 0.1889 seconds, totaling 1.5749 seconds in the recorded runs, excluding fixture creation and flake locking. These are local timings, not a calibrated evaluator performance guarantee.

Final consistency check: all 51 included probe commands were replayed after assembling this report. Every exit code and stdout value matched the recorded result. The replay took 1.617 seconds locally. Stderr wording and trace locations were not compared as a stability contract.