Coding agents can be evaluated. We just have to evaluate the work.

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

I recently argued with a software factory provider, whose position was that coding agents cannot be evaluated.

Their reasoning was understandable. Software engineering is open-ended. Requirements are incomplete. Repositories contain years of undocumented decisions. Two engineers can solve the same problem in completely different ways, and both can be right. An agent may fail one run and succeed on the next. A benchmark can never reproduce all the context, negotiation, and judgment involved in shipping production-grade software.

“Coding agents are non-deterministic, operate over long horizons, and can produce many valid solutions. None of that makes them unevaluable. It means we need to stop grading them like chatbots.”

All true. But “difficult to evaluate” and “cannot be evaluated” are very different claims.

We already evaluate traditional software systems with many possible implementations. I argue coding agents should be held to the same standard.

Agent !== Model

A coding agent is not just a model.

It is made up of a model, a harness, tools, repository context, instructions, permissions, an execution environment, and a feedback loop. Change any one of those and the outcome can change materially. A stronger model with poor repository context may perform worse than a smaller model with the right tools and a fast test suite. 

This is one reason public benchmarks are easy to misuse. A score is often described as if it measures the underlying model, when it actually measures a particular model-agent-environment combination under a particular token and time …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://thenewstack.io/evaluating-coding-agents-framework/
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.