PROBE Benchmark Shows Coding Agents Fail to Find Real Bugs
October 28, 2026
Evaluation of frontier models on the PROBE benchmark, covering 2,457 GitHub bug fixes, reveals that coding agents achieve less than 21% recall and under 9% precision. While stronger models increase recall from 8% to 17%, precision does not improve, even though 85% of findings are judged valid by humans.
HOW THIS AFFECTS YOU
●
builderDo not rely solely on current frontier models for high-precision automated bug discovery in large repos.
●
researcherThis highlights a massive gap between agent-detected issues and actual maintainer-fixed bugs.