Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluation

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2601.15161v3 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety. These risks are hard to assess: subtle clinical errors are often missed by generic metrics and LLM judges using general criteria, while expert-authored fine-grained rubrics are expensive and difficult to scale. In this paper, we propose a retrieval-augmented multi-agent framework for automatically generating instance-specific evaluation rubrics. Our approach grounds evaluation in authoritative medical evidence by decomposing retrieved content into atomic facts and synthesizing them with user interaction constraints to form fine-grained evaluation criteria. Evaluated on HealthBench and LLMEval-Med, our framework achieves Clinical Intent Alignment (CIA) scores of 50.20% and 31.90%, significantly outperforming the GPT-4o baseline and showing consistent improvements across English and Chinese medical benchmarks. In discriminative tests on HealthBench, our rubrics achieve a 7.8% point higher win rate than GPT-4o and increase the mean score difference from 4.972 to 8.658. Ablation studies further show that individual components contribute differently across datasets, with interaction-intent modeling providing the most consistent contribution to clinical-criterion coverage. Beyond evaluation, our rubrics guide response refinement, improving response quality by 9.2%. These results suggest that automated, knowledge-grounded rubric generation provides a scalable …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2601.15161
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.