Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2605.18490v2 Announce Type: replace Abstract: We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: a single-round Vector RAG system and an LLM-compiled markdown wiki browsed by a tool-using agent. Both systems answered the same 13 questions over 24 papers using the same answer-generating model, and their answers were scored by two blinded LLM judges. The three preregistered predictions, in registered order, came out one weakly supported, one supported, and one refuted. The wiki was predicted to synthesize better across papers; it scored much better at connecting findings, but its organization advantage fell below the registered threshold once both judges' scores were combined. RAG was predicted to hold its own on single-fact lookup, and it met the registered test, though the second judge alone would have refuted it. The wiki was predicted to be expensive to build and cheap to query; the build side held by roughly two orders of magnitude, but the query side reversed: the wiki spent about 21 times more tokens per query, so no break-even point exists. Two exploratory analyses explain the disagreement. A decomposition-retrieval variant of RAG removes almost all of the wiki's synthesis advantage at lower token cost, though not its advantage in claim-by-claim citation support. Holistic groundedness scoring disagrees with atomized citation checking by direction, and between judges: rank agreement on that criterion is near zero (rho = 0.04), against rho = 0.81 on the most concretely defined criterion. Grounded research …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2605.18490
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.