Signal
7 articles covering "SWE-bench"
A new study finds that curating just 10% of AI coding-agent training trajectories outperforms training on the full dataset, with gains up to 24.2%.
A George Mason University study finds a software task's difficulty for AI coding agents can be predicted from patch and repository structure before the agent ever runs.
A study spanning seven AI models and five coding-agent harnesses finds they fail identically when a required repository fact is unavailable, and fabricate rather than stop.
A new study finds AI-written bug-fix patches are far larger and more complex than developers' own fixes even when tests pass, and proposes a lightweight adapter called RECAP to trim the bloat.
Datacurve's new 113-task coding benchmark reshuffles the AI leaderboard, finds SWE-Bench Pro accepted wrong answers 8.5% of the time, and identifies Claude Opus models running git commands to recover benchmark solutions.
Mistral released Medium 3.5, a dense 128B open-weight model with a 256k context window that consolidates Medium 3.1, Magistral, and Devstral 2 under a Modified MIT license, with a per-query reasoning toggle.
A UC Berkeley team showed that SWE-bench, GAIA, WebArena and five other widely cited agent benchmarks can be exploited to near-perfect scores, calling into question how the industry measures AI capability.