Source
docs/research-derived-tasks.md
Static HTML generated from the repository markdown.
Guide
Research-Derived Tasks
These tasks were added from community reports about how LLMs mishandle Nix and NixOS work. The source context is also included in each task prompt so agents and benchmark readers can see what real complaint the task models.
These tasks were added from community reports about how LLMs mishandle Nix and NixOS work. The source context is also included in each task prompt so agents and benchmark readers can see what real complaint the task models.
This list should be refreshed weekly from new NixOS Discourse issues, especially threads that show concrete agent failures, incorrect advice, maintainer corrections, or novel Nix/NixOS constraints that fit a deterministic benchmark.
| Task | Modeled complaint | Source context |
|---|---|---|
module-stale-option-migration | LLMs suggest deprecated or removed NixOS option paths from older releases. | Reddit: https://www.reddit.com/r/NixOS/comments/1k0s6co/nixos_llms_is_really_exciting/ and https://www.reddit.com/r/NixOS/comments/1dkgkso/best_language_model_for_nix_language_questions/ |
package-name-lookup-contract | LLMs make up package names or suggest names that are absent from the available package set. | Reddit: https://www.reddit.com/r/NixOS/comments/1rqy6h0/nixos_migration_was_relatively_easy_using_llm/ and https://www.reddit.com/r/NixOS/comments/1n6er8g/nixos_for_a_beginner_in_2025/ |
container-native-vs-oci | LLMs mix native NixOS containers with Docker/Podman-style container abstractions. | Reddit: https://www.reddit.com/r/NixOS/comments/1ga6dbe/examples_of_questions_you_asked_an_llm_about_nix/ |
string-escaping-systemd | LLMs often mishandle Nix string escaping when shell code must keep runtime variables literal. | NixOS Discourse: https://discourse.nixos.org/t/local-ai-agent-for-nixos-configuration-looking-for-experience-reports/77238 |
debug-network-false-lead | LLMs chase plausible diagnostic rabbit holes instead of following observed evidence. | NixOS Discourse: https://discourse.nixos.org/t/nixos-hard-locks-driving-my-crazy/72210 and https://discourse.nixos.org/t/broken-state-of-the-system-after-25-11/73249 |
python-cuda-uv2nix-patch | LLMs struggle with NixOS flakes, Python-on-NixOS, CUDA packaging, and suggest generic Linux fixes. | NixOS Discourse: https://discourse.nixos.org/t/setting-deep-learning-python-project-with-cuda-support-using-uv2nix/72028 |
issue-report-quality | AI-generated reports can be low-signal, unverified, and costly for maintainers to triage. | GitHub: https://github.com/NixOS/nixpkgs/issues/410741 and NixOS Discourse: https://discourse.nixos.org/t/proposal-to-have-an-ai-usage-policy/75650 |
home-manager-extra-special-args | LLMs often pass flake inputs into nixosSystem and assume Home Manager modules receive them automatically, missing the need for home-manager.extraSpecialArgs. | NixOS Discourse: https://discourse.nixos.org/t/pass-specialargs-to-the-home-manager-module/33068 and https://discourse.nixos.org/t/infinite-recursion-that-i-cannot-figure-out/71077 |
home-manager-wsl-module-import | LLMs fail to distinguish Home Manager's standalone entrypoint from importing Home Manager as a NixOS module, especially in WSL setups. | NixOS Discourse: https://discourse.nixos.org/t/cant-enable-home-manager-in-wsl-nixos/49364 |
home-manager-xdg-files | LLMs suggest imperative directory creation, symlink farms, or shell snippets for home folders and $HOME/.config files instead of Home Manager's XDG/file options. | Reddit: https://www.reddit.com/r/NixOS/comments/175a3e6/is_it_right_chatgpt_how_to_create_the_documents/ and NixOS Discourse: https://discourse.nixos.org/t/download-config-files-to-home-dir/60934 |
flake-input-package-selection | LLMs assume every flake package set has packages.${system}.default and miss the actual named package output. | NixOS Discourse: https://discourse.nixos.org/t/error-attribute-default-missing/59474 |
module-system-boundaries | LLMs blur NixOS, Home Manager, and nix-darwin module systems instead of sharing plain data and writing separate modules. | Reddit: https://www.reddit.com/r/NixOS/comments/1gwgzbd/nixos_on_macos_nix_not_picking_up/ and https://www.reddit.com/r/NixOS/comments/1rsassa/welcome_to_den_v0120/ |
overlay-module-boundary | LLMs and generic advice sometimes misuse overlays as if they could also define module options such as systemd.user.*, instead of separating package overrides from module config. | NixOS Discourse: https://discourse.nixos.org/t/overlay-with-user-systemd-unit-and-timer/9295 |
fhs-binary-wrapper | Generic Linux or LLM advice suggests creating global FHS directories on NixOS instead of wrapping, patching, or using AppImage/FHS tooling. | Reddit: https://www.reddit.com/r/NixOS/comments/17tw9ik/filesystem_hierarchy/ and NixOS Discourse: https://discourse.nixos.org/t/newbie-question-about-running-appimage/41564 |
mutable-config-home-manager | LLMs suggest managing mutable GUI application profile files as read-only Home Manager files, which breaks apps that need writable state. | NixOS Discourse: https://discourse.nixos.org/t/purpose-of-authentications-option-home-manager/76999 and https://discourse.nixos.org/t/strategies-for-declarative-approaches-to-programs-with-mutable-configuration-files/66276 |