Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2602.16111v2 Announce Type: replace-cross Abstract: Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based labeling provides a high-fidelity reference measurement, but is cost-prohibitive to run per experiment, per arm, per segment, and per day on a platform with hundreds of concurrent experiments. We describe a surrogate-based prevalence measurement system deployed in Pinterest's experimentation platform. The contribution is system-level rather than estimator-level: the system maintains a single global calibration of ML score buckets, continuously refreshed from a recurring LLM-labeled stream, and reuses the resulting bucket-level prevalences across every experiment via a per-experiment SQL metric and a delta-focused dashboard. Because the calibration is derived from the platform's daily-batch prevalence samples, it remains representative of production traffic as distributions drift, and in Pinterest's deployment it incurs zero incremental labeling cost. Teams without such infrastructure can instantiate the same pattern with a recurring calibration-labeling workflow whose cost is amortized across all downstream experiments rather than paid per experiment, arm, segment, and day. The system serves ~100 experiments and ~250 arms per day across six calibrated content categories, including a holdout program. Relative to per-experiment LLM labeling, which in practice yields a one-shot read per arm on a small subset of experiments, the surrogate …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2602.16111
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.