Researchers Show Coding-Agent Task Difficulty Is Predictable From Static Code Structure Alone, Hitting 0.863 AUC

Overview

AI-based coding agents are “rapidly advancing, with the best open-source systems now resolving more than half the tasks on SWE-bench Verified,” but according to a new study from George Mason University, “the aggregate solve rate…provides no account of why a task is hard” (arXiv). Researchers Ebtesam Al-Haque and Brittany Johnson, both of GMU’s Department of Computer Science, set out to fix that gap, and found that a task’s difficulty for an AI agent can be predicted before the agent ever attempts it — using nothing but structural properties of the code patch, the repository, and the issue description itself, according to the paper posted to arXiv on August 18, 2026 and accepted to the 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026), to be held in Munich, Germany on October 8–9, according to the paper (arXiv).

What We Know

The paper, titled “What Makes Software Issue Resolution Tasks Difficult for Agents?”, builds a measurement framework around CoderForge-Preview, which the authors describe as “the largest open dataset of coding agent trajectories to date, consisting of 258K test-verified trajectories spanning 51K tasks across 1,655 repositories,” drawn from three existing benchmark sources: “R2E-Gym (4,216 tasks), SWE-Smith (37,221 tasks), and SWE-Rebench (9,764 tasks)” (arXiv). The trajectories were generated by running “Qwen3-Coder-480B, one of the best-performing open-source models on agentic coding benchmarks, operating within an OpenHands v0.52.1 scaffold where the agent iteratively issues bash commands and file edits for up to 100 steps within an isolated Docker container,” according to the paper (arXiv).

From that dataset, the researchers extracted 54 features spanning the task’s patch, its repository, and its natural-language prompt, then tested how well those static, pre-run features could predict two outcomes: whether the agent succeeded at all, and what fraction of its runs passed all tests. The headline result: “task difficulty is substantially predictable from static features (AUC=0.863)” — far above a majority-class baseline, which “achieves AUC=0.500 and MCC=0.000, confirming that it carries no discriminative information,” the paper states (arXiv). The two top-performing classifiers, XGBoost and Random Forest, reached “near-identical performance on any_success (XGBoost: AUC=0.863, MCC=0.549, Brier=0.129; Random Forest: AUC=0.863, MCC=0.538, Brier=0.128),” while a regression model predicting the continuous pass rate reached an R² of 0.408, according to the paper (arXiv).

Notably, patch features alone — without any information about the repository or the prompt — “achieve AUC=0.846 on any_success and R²=0.375 on pass_rate,” according to the paper, meaning most of the predictive power comes from how the code change itself is shaped, not from the surrounding project (arXiv). Using SHAP attribution — a method for ranking how much each feature contributes to a model’s predictions — the researchers found that “the three most important features were patch_lines_deleted, patch_num_hunks, and patch_hunk_gap_mean (ranks 1–3),” with patch_lines_deleted alone carrying “a mean absolute SHAP of 0.406—2.65× larger than the fourth-ranked feature (repo_top_level_dir_count, 0.153).” Together, the top three features “account for 29% of total mean absolute SHAP across all 54 features,” and the pattern is consistently negative: “high feature values push predictions toward failure,” the paper reports (arXiv).

In plain terms, tasks whose correct fix involves more deleted lines, more separate hunks of changed code, and hunks that are spread further apart in a file are harder for the agent to resolve — a pattern the authors summarize as difficulty being “largely driven by patch fragmentation.” Repository scale is the second major driver: the paper defines “scale features (file count, codebase size, directory count)” as reflecting “the size of the search space the agent must traverse,” alongside directory-structure features such as “root-level breadth, maximum and mean nesting depth” that reflect “the structural complexity of that space,” with the scale subconstruct built from five specific features — “total files, Python files, total codebase size in bytes, mean file size, and total directory count” (arXiv).

Properties of the issue-report text itself, rather than the code, matter most for tasks in the middle of the difficulty spectrum. The researchers found that “at least one prompt feature ranks among the five largest contributors for 70.3% of near-baseline tasks (404/575), 26.8% of easy tasks (239/893), and 6.8% of hard tasks (60/886),” with the most frequent contributors involving “referential, coordination, and attachment ambiguity markers derived from computational linguistics theory” — that is, wording in the issue description that is ambiguous about what it refers to, how clauses relate, or which words modify which (arXiv). The two outcome variables the study predicted are defined precisely: any_success is “an indicator of whether at least one run succeeded (positive rate: 67.5%),” and pass_rate is “the fraction of runs that passed all tests (continuous, [0,1]; mean=0.593, SD=0.455),” according to the paper (arXiv).

Why It Matters for Developers

The paper frames its findings around a specific practical problem with how coding-agent benchmarks are currently read: a single aggregate solve-rate number obscures whether an agent is good at everything or only at the easy fraction of a benchmark. The authors argue that “a developer who has a sense of how structurally complex a task is can look at an agent’s performance within the corresponding difficulty band on a benchmark, rather than its overall solve rate, to form a more accurate expectation” (arXiv). Because the predictive features are all static — derivable from the patch, repository, and prompt without running any model — the difficulty score can, in principle, be computed for any task before an agent is ever pointed at it.

The same logic applies to benchmark design itself. “Computing difficulty scores for each task in an existing benchmark reveals whether the difficulty distribution is structurally balanced or concentrated in a narrow region,” the researchers write, pointing toward difficulty-controlled or difficulty-stratified benchmark construction as a downstream use of the framework (arXiv). The paper’s conclusion frames this as part of a broader shift: “static, deterministic features derived from the gold patch, repository, and issue prompt predict agent success with AUC=0.863 without any model inference. This enables pre-hoc difficulty estimation and opens the door to structurally controlled benchmark construction, difficulty-stratified evaluation, and more grounded developer reliance on agent tools” (arXiv).

What We Don’t Know

The study evaluated trajectories from a single agent configuration — Qwen3-Coder-480B running in the OpenHands v0.52.1 scaffold — so it is not yet established whether the same static features predict difficulty equally well for other models or agent harnesses. The authors acknowledge this directly, stating that “future work will extend the framework across multiple agents to expose per-agent difficulty profiles and, in aggregate, a global difficulty landscape that characterizes the structural frontier of agentic software engineering” (arXiv). The paper also does not report how the difficulty-prediction framework performs on code changes that are not represented in CoderForge-Preview’s three source benchmarks, and as a peer-reviewed conference paper still awaiting its October presentation, the findings have not yet been through the ESEM 2026 review discussion.

Analysis

The result reframes what “benchmark saturation” means for coding agents. As open models increasingly clear the majority of SWE-bench Verified tasks, a single solve-rate percentage tells a developer less each year about where an agent’s actual limits are. A pre-hoc difficulty score derived purely from patch and repository structure — computable in the time it takes to inspect a diff, with no agent run required — offers a cheaper way to route work: routine, low-fragmentation fixes to an agent, and multi-hunk changes scattered across a large codebase to a human, or to a more careful review pass. The authors’ own framing extends this beyond issue resolution: they write that “the measurement framework generalizes naturally to other agentic SE tasks such as feature implementation, test generation, where analogous structural properties of the target artifact and the natural-language specification can be represented as static difficulty signals” (arXiv) — suggesting the same patch-fragmentation and repository-scale signals could eventually inform triage decisions well beyond bug fixes.