[arXiv]score: 0.28

Coding Agents Submit Patches Confidently but Resolve Far Fewer: GPT-5 at 44%, Llama 4 at 18%

June 23, 2026

Across 1,750 trajectories on 50 SWE-bench Verified tasks, GPT-5 submits patches on 100% of runs but resolves only 44%; Llama 4 submits 99% but resolves 18%; Gemini submits 70% but resolves 50%. The dominant failure mode is silent semantic failure, where agents repeatedly submit plausible-looking but incorrect patches, accounting for 80% of Llama 4's failures.

HOW THIS AFFECTS YOU

●

builderSubmit rate is a misleading proxy for agent reliability — you need test-verified resolve rate as a production metric, and should design for silent semantic failure detection.

●

researcherSilent semantic failure — repeated identical misinterpretations rather than random errors — is a distinct failure mode that current benchmarks undercount and needs targeted evaluation.

●

founderGemini outresolves GPT-5 despite submitting 30% less often, which matters for product decisions around which model to deploy in agentic coding workflows.

read original ↗arxiv.org

DAILY DIGEST

you don't check 9 sources — we do. one email every morning, read in 2 min. free. unsubscribe anytime. privacy

← back to feed