Artificial Intelligence · 31.08.2026, 23:47 UTC
Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour
| Schweregrad | info |
|---|---|
| Kategorie | Artificial Intelligence |
| Quelle | MarkTechPost ↗ |
| Veröffentlicht | 31.08.2026 UTC |
Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.
How do you benchmark a web search API when the thing being tested can read the answer key? A search agent has a fetch tool. If the gold labels sit in a public dataset, the agent can download them mid-evaluation and skip retrieval entirely. A similar problem arises when the answers are already encoded in the model’s parametric memory: a correct response no longer demonstrates that web search worked. Keenable’s answer is NEEDLE, a live open-source benchmark that rebuilds its query set from fresh public sources rather than freezing one. News queries are regenerated hourly from RSS feeds and Google Trends; finance, scholar, legal, and rare-entity queries are regenerated daily from SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs. Fifteen search APIs run against the same query text under one protocol, and every score is read against ultimate, a pooled oracle engine that marks what the whole field managed to find.
Is it reproducible?
Yes, as an open source evaluation harness rather than a product. needle is a Python CLI installed with uv sync and driven by two subcommands per benchmark, generate and run. It needs an OpenRouter key for judging and one API key per engine tested, and runs on a laptop or in CI. It allows recreated all query streams that are being used in addition to the ranking quality judgements.
What NEEDLE measures
NEEDLE stands for News, Everyday, Expert, Deep-tail, and Legal Evaluation. Each vertical models a different agent intent. News projects the newest item from ~124 curated RSS feeds and Google Trends into a keyword query. Finance …