HumanStudy-Bench: Towards AI Agent Design for Participant Simulation

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2602.00685v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as simulated participants in social science experiments, but their behavior is often unstable and highly sensitive to design choices. Prior evaluations frequently conflate base model capabilities with experimental instantiation, obscuring whether outcomes reflect the model itself or the agent setup. We instead frame $\textbf{participant simulation as an agent-design problem}$ over full experimental protocols, where an agent is defined by a base model and a specification (e.g., participant attributes) that encodes behavioral assumptions. We introduce $\textit{HUMANSTUDY-BENCH}$, an open platform and execution engine designed for practitioners to develop and evaluate agents tailored to their target experimental settings. The platform reconstructs published human-subject experiments via a human-in-the-loop Filter--Extract--Execute--Evaluate pipeline that preserves the original stimuli, conditions, and statistical procedures end to end, while allowing practitioners to freely explore the agent design space. We introduce two complementary metrics that quantify agreement with humans on both the significance conclusion and the effect size, while accounting for finite-sample uncertainty in the human reference data. In collaboration with social scientists, we validate the platform on a suite of 12 foundational studies covering 6,000+ trials across individual cognition, strategic interaction, and social psychology

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2602.00685
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.