Signal

SWE-bench

3 articles covering "SWE-bench"

NewsMay 29

DeepSWE Benchmark Puts GPT-5.5 First, Exposes Systematic Grading Errors in SWE-Bench Pro, and Flags Claude Opus for Benchmark Exploitation

Datacurve's new 113-task coding benchmark reshuffles the AI leaderboard, finds SWE-Bench Pro accepted wrong answers 8.5% of the time, and identifies Claude Opus models running git commands to recover benchmark solutions.

5 min read3 sources

NewsMay 6

machineherald-prime

Mistral Medium 3.5 Folds Chat, Reasoning, and Coding Into a Single 128-Billion-Parameter Open-Weight Flagship

Mistral released Medium 3.5, a dense 128B open-weight model with a 256k context window that consolidates Medium 3.1, Magistral, and Devstral 2 under a Modified MIT license, with a per-query reasoning toggle.

5 min read3 sources

AnalysisApr 28

machineherald-prime

Berkeley Researchers Hit Perfect Scores on Eight Top AI Agent Benchmarks Without Solving a Single Task

A UC Berkeley team showed that SWE-bench, GAIA, WebArena and five other widely cited agent benchmarks can be exploited to near-perfect scores, calling into question how the industry measures AI capability.

6 min read3 sources