Evidence of conceptual mastery in the application of rules by Large Language Models

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2503.00992v3 Announce Type: replace-cross Abstract: Background. Evidence that large language models (LLMs) reproduce human judgments does not establish conceptual mastery: the correspondence may reflect memorisation or be sensitivite to incidental task features. Objective. Across five experiments, we test whether 13 LLMs possess a generalisable competence in applying rules, including cases in which a rule's text and purpose point towards different outcomes. Method. Study 1A compared LLM judgments with newly collected human data on published stimuli and matched vignettes created after the models' training cut-offs. Studies 2A/2B compared responses to time-pressure instructions, a manipulation with a mechanistic route to human judgment blocked for LLMs. Study 3 varied reasoning effort, as an analogue for time constrained human judgements. Studies 1B/2B alsovaried system prompt wording and numerical scale anchors. Results LLM judgments closely tracked human judgments for both stimulus sets, while responding in the same unanticipated purposivist direction in the new set as humans did. Sensitivity to text and purpose was robust across prompt variations. Responses to time-pressure instructions were model-specific, suggesting a distinction between conceptual competence and human alignment. Replication of the human pattern was most apparent in models with fewer parameters, and these effects were susceptible to prompt variation. Increasing reasoning effort produced no detectable change in rule application for most models though a significant purposivist trend was observed …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2503.00992
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.