Evidence Over Anecdotes: Running A/B Tests on AI Agent Tooling by Sarah Martinelli Benedetti

Sicherheitsmeldung mit Schweregrad hoch. Sie wird laut CISA-KEV aktiv ausgenutzt und sollte priorisiert behandelt werden. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

AI agents are non-deterministic: run the same task twice with zero code change, and the model’s own sampling can produce a different outcome. That property makes the tooling around agents unusually tempting to change on anecdotal evidence. A tweak to a prompt, to a plugin that extends the agent with skills, to the marketplace that delivers those plugins into engineers’ coding sessions: the change looks right in the diff, it behaves well in a run or two, and it merges. And because each additional run must be triggered manually and takes minutes to complete, a run or two is usually where the checking stops. For ordinary code, that evidence standard wouldn’t survive review. We’d ask for tests. We’d ask what happens on the unhappy path. Agent tooling deserves the same skepticism, and arguably more, precisely because of the non-determinism that makes a single good run mean so little. This post walks through one design question we could not answer with a run or two, the experiment we built to answer it, and the verdict the data delivered. Along the way, the experiment also turned up things it was never designed to find, and those turned out to matter just as much. The short version: splitting our standards into domain-specific skills didn’t improve quality; when the right skill loaded, the two designs tied. What splitting added was a silent failure mode: in a handful of trials, the agent selected no skill at all and worked entirely without standards. Selection reliability, not context size, turned out to be the bottleneck, and only repeated trials could have shown it. The rest …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://www.pagerduty.com/eng/evidence-over-anecdotes-running-a-b-tests-on-ai-agent-tooling/
• Priorität HOCH: laut CISA-KEV aktiv ausgenutzt – bevorzugt patchen.
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.
• Erkennung: Logs und Indikatoren (IoCs) auf Ausnutzung prüfen.