Artificial Intelligence · 01.08.2026, 10:03 UTC
Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks
| Schweregrad | info |
|---|---|
| Kategorie | Artificial Intelligence |
| Quelle | MarkTechPost ↗ |
| Veröffentlicht | 01.08.2026 UTC |
Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.
Supabase has open sourced Supabase Evals, its benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks, such as building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, then scores the result. It powers the public leaderboard at supabase.com/evals and an internal regression suite monitored daily.
Is it deployable?
Yes, today. supabase/evals is public under Apache-2.0 and runs locally via pnpm.
Industries: Developer tooling, cloud infrastructure, data platforms, and regulated backends in fintech or healthcare, where an agent writing a wrong RLS policy is a security incident.
Applications: Regression-testing docs and skill edits, gating SDK releases, and comparing agent harnesses head to head.
Constraints: Local-stack runs need a Docker daemon, provider API keys, and ports 54321–54329 free.
How the harness works
Supabase defined three dimensions: products (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), topics (RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema), and stages (build, deploy, investigate, resolve). It then picked the smallest scenario set touching each dimension once, grounded in support tickets, bug reports, and GitHub issues.
Scenarios split into two suites. Benchmark scenarios cover breadth and are published. Regression scenarios cover known failure modes, refresh daily, and do not move published scores.
Every scenario runs against a real …