DevOps / SRE / Platform · 21.08.2026, 12:01 UTC
Most coding agent benchmarks skip large-scale refactoring. Not this one.
| Schweregrad | info |
|---|---|
| Kategorie | DevOps / SRE / Platform |
| Quelle | The New Stack ↗ |
| Veröffentlicht | 21.08.2026 UTC |
Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.
AI coding agents still struggle with large-scale refactoring, with the best model achieving only a 41.2% resolve rate on a new refactoring-focused benchmark developed byresearchers at Shanghai Jiao Tong University, Peking University, Douyin Group, and other institutions.
It’s no secret that AI benchmarks aren’t perfect. A recent audit cited by the researchers shows how misleading AI coding agent benchmarks can be, with nearly 60% of unsolved SWE-bench Verified instances containing flawed tests.
“We don’t have LLMs that would be able to read a large codebase and see it in its entirety and ‘understand’ it all at once,” he says. “That level of capability is still far out of reach.”
For AI coding agents, specifically, evaluation quality is considered to be on the decline, as frontier models can end up sailing through benchmarks if solutions have leaked into training sets. But when models can manage impressive-looking scores without really doing all the work, it raises the question of whether those benchmarks are truly a good indication of real-world agent capabilities.
As Vojtěch Pavlík, senior director of technical strategy, core infrastructure, SUSE, tells The New Stack, many sought-after capabilities are still a ways away:
“We don’t have LLMs that would be able to read a large codebase and see it in its entirety and ‘understand’ it all at once,” he says. “That level of capability is still far out of reach.”
A new benchmark attempts to clear things up. SWE-Bench ProMax is a multilingual code refactoring benchmark of 170 instances from real commits across seven …