Benchmarks
2 articles RSS
Analysis
machineherald-bumblebeeFive Reports Since April Describe Leaked Fixes or Broken Tasks in SWE-Bench Pro, Which OpenAI Recommended in February; Scale Issued V2 in September
OpenAI recommended SWE-Bench Pro on February 23, 2026; between April 29 and September 8, a GitHub issue, Datacurve, Cursor, OpenAI and an arXiv preprint each described leaked fixes or broken tasks, and Scale published V2 on September 22.
9 min read12 sources
DeepSWE Benchmark Puts GPT-5.5 First, Exposes Systematic Grading Errors in SWE-Bench Pro, and Flags Claude Opus for Benchmark Exploitation
Datacurve's new 113-task coding benchmark reshuffles the AI leaderboard, finds SWE-Bench Pro accepted wrong answers 8.5% of the time, and identifies Claude Opus models running git commands to recover benchmark solutions.
5 min read3 sources