Researchers Coin 'Coherence Debt' to Explain Why Coding Agents Fabricate Fixes Instead of Asking When Facts Go Missing
A study spanning seven AI models and five coding-agent harnesses finds they fail identically when a required repository fact is unavailable, and fabricate rather than stop.
Overview
A new paper posted to arXiv on August 17 argues that coding agents fail on repository-scale edits for a specific, measurable reason: a required fact — a renamed function, a test’s expectation, a migration rule — is available to neither the agent’s context window nor the underlying model’s training. The authors call the resulting shortfall “coherence debt,” and define it as “at each edit, a required fact comes from recent context or parametric memory, and the facts covered by neither form coherence debt”. Testing seven model families across five agent harnesses, the paper — titled “The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks” and authored by Bardia Mohammadi, Lars Klein, Aman Chadha, Akhil Arora, and Laurent Bindschaedler — finds that “a missing fact produces wrong work rather than absent work: an agent asked to act acts, fabricating the file or guessing the value”, rather than stopping to ask.
What We Know
The researchers ran controlled experiments that alternately supplied and withheld two “channels” a coding agent can draw a fact from: recent context (what the harness put in the prompt or the agent read) and parametric memory (what the underlying model already knows from training). On four hand-authored fictional API workloads, “with no workspace and no task-specific prior, all 154 novel closed-book trials score 0/12: no family guesses even one complete migration”, with a 95% statistical upper bound of just 2.4% on any nonzero success rate. Putting the same facts directly in the prompt reversed that completely: “putting the same facts in the prompt lifts 299 of 300 matched trials to at least 9 of the 12 requirements”.
A second experiment tested what happens when a model’s memorized knowledge is deliberately defeated. The team took a real library, Pydantic, and renamed its surface forms so a model’s training-derived knowledge would actively mislead it. The result was strikingly uniform: “on the renamed migration, 66 of 70 trials across seven models end at the same score, each passing the identical 24 of the 79 tests” — meaning seven different model families, tested independently, converged on the exact same failure point.
The paper also tested whether where a fact sits in the context window matters as much as whether it’s there at all. It found distance was irrelevant: a “supplied fact is used as reliably at the far end of a 128,000-character context as beside the edit”, while withholding a fact “costs exactly the work they support and no more,” with damage falling linearly rather than compounding, according to the paper.
On cost, the study compared harnesses — Claude Code, Codex CLI, Aider, OpenHands, and opencode — on configurations that all passed every test in a task. Despite identical outcomes, “the tokens consumed over a run differ by 12.8× while the amount held in front of the model at any moment differs by only 1.8×, and the expensive configurations recover nothing extra when facts are withheld”. In raw numbers, cumulative token spend ranged from 293,882 to 3,752,134 tokens across configurations that all reached the same result, with the cheapest run completing in five tool calls and the most expensive in seventy-nine, according to the paper’s full text. The gap wasn’t explained by extra reasoning effort either — “reasoning tokens are at most 0.46% of input and explain none of it”.
A recurring theme is that agents rarely say when they’re stuck. The paper reports wide variation in how often models explicitly flag a missing fact instead of guessing: in one block of trials pairing specific models with specific harnesses, Opus reported being blocked in 100% of trials, compared with 75% for a model the paper calls Fable, 25% for Sonnet, 12.5% for Haiku, and 0% for both GPT-5 paired with Codex CLI and GLM-5.2 paired with opencode, for a pooled rate of 37% across 46 trials, according to the paper’s tables. As the authors put it, “how often it says it is blocked instead is a property of the model, from every trial to none”.
Availability of a fact doesn’t guarantee a correct edit, either. When a written coding standard and the actual working code stated the same fact differently, “agents follow the standard in every one of 39 trials across two harnesses, even where that means writing the worse code” — leading the authors to conclude that “a stale convention file costs more than no file at all”.
The paper additionally raises a validity concern about SWE-bench Verified, a widely used benchmark for coding-agent capability. Because many benchmark repositories are likely represented in the models’ training data, the authors argue that “because parametric memory substitutes for reading, on SWE-bench, where models likely know the repositories, reads no longer predict success”. Running 100 SWE-bench Verified instances across eight repositories (400 attempts, 397 scored), the team found that a “residency” score meant to measure whether an agent is actually reading the facts it needs produced “within-cell AUC ≈0.49, which is chance” — no better than a coin flip — once an earlier, statistically leaky version of the metric was corrected. Resolved rates on that same SWE-bench slice ranged from 1.0% for Sonnet paired with Aider up to 39.4% for GPT-5 paired with Codex CLI, according to the paper.
What We Don’t Know
The paper is a preprint and has not been through peer review; as of publication it had not yet drawn any secondary press coverage or public discussion outside arXiv itself. The authors themselves flag limits on how far the results generalize: several of the mechanism-probe experiments — including ones separating harness policy from model capacity — ran on small sample sizes (as few as three trials per cell) and are described in the paper as exploratory rather than headline findings. The four fictional API workloads used to isolate the closed-book/front-load effect were also hand-authored specifically to avoid any training-data leakage, so it isn’t yet established how cleanly the same numbers would reproduce on a broader set of real-world libraries and languages beyond the Pydantic-based experiments.
Analysis
The practical thrust of the paper is aimed squarely at teams building or operating coding-agent harnesses rather than at model vendors. The authors’ core prescription is that “harnesses should keep the facts an edit depends on available when the agent writes, and check that availability against what the agent produces rather than what it reads” — a distinction that matters because their data shows agents will confidently produce output even when the facts behind it are missing, which means simply logging what a harness fed into a model or what an agent chose to read is not a reliable signal of whether the edit is actually grounded. Combined with the token-cost findings, the paper’s broader implication for tool builders is that larger context windows and higher spend are not, by themselves, a fix: two harnesses reaching identical outcomes differed more than tenfold in tokens consumed, which the authors attribute to how often a harness has to rebuild the same working set rather than to any difference in reasoning effort.