Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2606.21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories. However, scaling OPD to long-horizon reasoning exposes a reliability and efficiency problem: standard OPD assigns every candidate the same long rollout budget, even though some trajectories may quickly become weakly aligned with the teacher and provide less useful supervision. Prior analyses suggest that teacher--student compatibility is important for OPD success, motivating early-prefix top-k overlap as a proxy for continuation value. Continuing low-overlap trajectories to the full rollout length may cause them to drift further from the teacher as generation proceeds, increasing computational cost while providing limited distillation benefit. To address this, we introduce Prefix-Guided On-Policy Distillation (PG-OPD), a rollout-allocation framework that estimates continuation value from fixed-length prefixes. PG-OPD computes teacher-student top-k overlap in an early probe window and allocates long rollouts only to high-overlap candidates, while stopping the rest at the prefix length. Across teacher--student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD achieves up to a 4.80-point accuracy gain and up to a 2.46x wall-clock speedup across configurations. These results demonstrate that using early-prefix compatibility to guide candidate pruning can improve both training efficiency and reasoning performance.

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2606.21994
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.