Models on the SWE-bench Verified leaderboard exhibit deletion avoidance, where they retain old code rather than removing it. When tested with strict removal requirements, frontier model performance dropped from 63.2% to 41.9% due to a tendency to wrap code in guards instead of deleting it.
HOW THIS AFFECTS YOU
●
builderBe cautious when using LLMs for code refactoring, as they may introduce technical debt by preserving obsolete code.
●
researcherThis identifies a specific failure mode in code-editing benchmarks that requires more rigorous evaluation.