Do AGENTS.md files actually help? What the benchmark found

The study below asks one narrow question: does a repository instruction file help a coding agent finish real tasks? The study's agents read the file under the name their tool uses — `AGENTS.md` for Codex and Qwen Code, and CLAUDE.md for Claude Code, which also reads AGENTS.md directly when no CLAUDE.md is present. The pitch from the tools that read them is simple: write down the build command, the house rules, and a map of the repository, and the agent will do better work.

The most direct controlled test so far says that when the file loads, it is usually followed, but it does not generally change whether the task succeeds, and it makes the run more expensive. A team at ETH Zurich and LogicStar tested four agent/model pairs on two sets of real GitHub issues. The paper is *Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?* (arXiv:2602.11988v3, revised 29 September 2026). Its abstract says providing context files "does not generally improve task success rates, while increasing inference cost by over 20% on average."

If you have seen the claim that AGENTS.md files are useless, the finding is more qualified: the file is followed, and it can still be worth writing for non-obvious rules.

Checked against each tool's documentation on 27 October 2026.

What the study measured#

The paper's setup is four agent/model pairs, each fed the file name its tool uses:

Each pair ran two benchmarks. SWE-bench Lite is 300 tasks from 11 popular Python repositories; none of those repositories had a developer-written context file, so the study used LLM-generated ones there. CTXbench is the study's own set: 138 tasks from 12 smaller Python repositories that already had a developer-committed context file. In each case the agent got no file, a file generated by the tool's own init command, or, on CTXbench, the file the repository's developers wrote. Success meant the agent's patch passed the tests for that issue. The study ran each task once per setting; cost is the model-call cost for that task.

What it found: more work, about the same success#

Compared with the same agent on the same task with no file, the averages in v3 were:

A step is one interaction with the environment, such as a shell command or a file edit. The success changes were not statistically significant. The step and cost increases were. The one significant success comparison was between the two kinds of file: developer-written files beat LLM-generated ones (p = 0.038).

The cost result is the stable one. The agents took more steps and ran more tests, and the paper says all context files "consistently increase the cost and number of steps required to complete tasks." A file is not free just because it is short.

The paper has been revised since it first appeared in February 2026. The February version said context files "tend to reduce task success rates"; the September version says providing them "does not generally improve task success rates", with developer-written files slightly ahead. The cost finding survived both. Treat the percentages as this version's averages, not as a fixed tax on your repository.

Why following the file did not solve more tasks#

The instructions were followed#

This is not a story about an agent ignoring the file. When a context file mentioned uv, the agents used it 1.6 times per instance; when it did not, they used it fewer than 0.01 times. Repository-specific tools were used 2.5 times per instance when mentioned, against fewer than 0.05 when not. The agents also ran more tests and explored more files. The paper concludes that the instructions were generally followed; the extra work is the cost.

The repository overview was the weak part#

Eight of the 12 developer-written files contained a dedicated codebase overview, a section describing the directory layout and what each part does. Every file generated by Sonnet 4.5 was flagged as containing one. The overview did not help the agent find the files that mattered: the number of steps before it touched a file in the real patch was not meaningfully lower with a context file, and for GPT-5.1 mini it was significantly higher.

The strongest evidence is an ablation, a follow-up run that removes one part to see whether it mattered. When the researchers removed the repositories' documentation files (Claude Code was excluded from that run for cost), LLM-generated context files improved performance by 2.7% on average and beat the developer-written ones. The generated overviews had been repeating documentation the agent could already read. Developer files still did slightly better than no file in the main run, though not significantly; that fits the paper's view that they add something beyond the documentation. The overview is the part that did not earn its tokens.

Length was not the explanation either. The paper found no strong relationship between file length and task success. A long file still spends tokens, the units the model reads and bills for, on every request, but the problem the paper identified was redundant content, not the line count. The size-limit guide covers the caps each agent enforces.

What to put in the file instead#

The study does not say delete every file. Its conclusion is narrower: context files should "only contain specific additional instructions beyond what is already available in the codebase." In practice:

1. Commands and tooling the agent cannot guess. The exact test command, the package manager, the generated-code step, the script that resets the local database. The paper found repository-specific tooling instructions were followed. Claude Code's docs list build commands and conventions as the things to write down, and Qwen Code's docs say "Don't include things Qwen can figure out by reading your code." Claude Code's docs also list project layout; the benchmark is the reason to keep that part short.
2. Traps and constraints, not a tour. "The files under db/legacy are generated; do not edit them" is a fact the agent cannot derive from the README. A directory-by-directory description is the part the benchmark found unhelpful.
3. Nothing the README or the docs already say. The benchmark's redundancy finding is the clearest cut: if the sentence is already in the repository's documentation, the agent can read it there.
4. Treat /init output as a draft, not the file you ship. /init is the command that asks the agent to generate a starter file from the repository. Claude Code's docs describe it as a starting point to refine "with instructions Claude wouldn't discover on its own"; Qwen Code's docs call its output a starter. The LLM-generated files in the study were the ones with the negative average, and most contained an overview. Generate, then cut.
5. Measure it on your own tasks. The paper's own conclusion is that any attempt to improve performance should be "rigorously evaluated before deployment." A simple version: pick five or ten real issues, run each with and without the file, and compare pass rate, steps, and cost. The averages hide per-repository differences; the paper found no single repository where the file had a significant effect.
6. For a rule that must hold, use permissions or hooks. Claude Code's docs say CLAUDE.md is context, not enforced configuration, and a PreToolUse hook is what blocks an action. A hook is a command the client runs at a fixed event, so it does not depend on the model deciding to obey. Other agents have their own enforced layers; the concept page covers the equivalents.

When the file is still worth it#

The benchmark measures one thing: whether the patch passes the tests. It does not measure whether the agent followed a style convention, treated a warning about a destructive command as a reason to stop, or needed the same correction twice. A short file can be worth its cost for those reasons. The paper's authors say context files are useful for "specifying non-standard coding practices"; that is a narrower claim than "better at coding."

It is a briefing, not a performance lever or a policy engine. If the sentence is a preference, put it in the file and accept that it is guidance. If the sentence is a boundary, put it in the layer the client enforces.

What the benchmark does not tell you#

• It is Python only. The paper says Python is well represented in the models' training data, so the model may already know the tooling; a niche language or an internal framework could produce a different result.
• It covers three agents, not every agent. Claude Code, Codex, and Qwen Code were tested. There is no controlled result here for Copilot CLI, Gemini CLI, or OpenCode; their pages cover what they load, not whether the file helps.
• It measures task success, not code quality or security. The paper names code efficiency and security as things it did not measure. A file can change behavior without changing the pass rate.
• It is one benchmark, and the models move. The numbers are v3's averages for the model versions in the table. The next model generation can change them. The shape of the finding — followed, costly, rarely decisive — is the part to keep.

Where to go next#

• What each agent reads, and where the file goes — the cross-tool map: AGENTS.md vs CLAUDE.md, load order, and what belongs in a file vs a skill.
• Claude Code loaded your CLAUDE.md but isn't following it — when the file loaded and still lost.
• Why Codex ignores part of your AGENTS.md — the chain and the 32 KiB cut.
• How long can AGENTS.md or CLAUDE.md be? — the limit each agent enforces.
• Context compaction — why a rule that loaded can fade in a long session.
• For the remaining per-tool pages: OpenCode, Copilot CLI, Gemini CLI, Qwen Code.