Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2608.02829v2 Announce Type: replace-cross Abstract: Model families are typically trained size by size, each from scratch. Can apretrained large model instead be converted into a smaller sibling? Wecharacterize the 1.4B->410M conversion in the Pythia family end to end.Representations align strongly across sizes (ridge R^2=0.84) while parametersalign weakly. Dense weight projection is functionally destructive, and abit-exact reconstruction control shows this is not an assembly artifact: basismixing breaks rotary, per-head, GELU, and LayerNorm structure. After the best-fitlinear operator, weight residuals are statistically indistinguishable from noiseunder shuffle controls. Conversion value therefore lives in initialization. Inmatched-budget continued pre-training we decompose conversion into twoindependent levers: least-squares compensation (a function lever, best zero-shot)and variance-preserving rescale (a dynamics lever, best endpoints). Compensationis a token-efficient, low-budget win rather than a universal one. At 30M tokens itbeats the strongest subcloning variant on both a width-reduced pair (84.0 +/- 1.8vs. 89.7 +/- 3.7, 3/3 seeds) and a held-out depth-reduced pair (109.3 vs. 117.9,3/3 seeds), reaching a given quality with fewer tokens. At a 33x larger budget thetwo converge to parity (40.0 vs. 40.0), both far ahead of from-scratch, whichtransfer initialization always beats: by up to 18x at low budget, with the marginnarrowing at convergence and at the largest scale. We also map the method'sboundary. At about 5x the donor scale (6.9B->1.4B) stacking both …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2608.02829
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.