ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2606.25156v3 Announce Type: replace-cross Abstract: Native length extrapolation remain a weakly solvable problem in language modeling due to trade-off balancing between exact retrieval fidelity, long-document likelihood, and inference efficiency. We present ATMA as a disciplined diagnostic for this specific problem. ATMA is a hybrid architecture recipe with modified Polar Attention that separates two stream of normalized direction channel and bounded participation-ratio magnitude channel, augmented with third recurrence channel which derived from gated-deltanet. We frame long-context modeling as Pareto problem than claiming general architectural dominance. We run 120-cell industrial ablation sweep for the first 1B-token training run to select promoted recipe, then continue with 10B-token training run for the second phase for more complete evaluation. We test 378M-parameter model from three attention variant (NoPE, RoPE, Polar) and Raven as external baselines from SSM-derived family for up to 256K length evaluation. Polar attention extends real-text retrieval to 64K length at 16.3% while both NoPE and RoPE variant collapse to 0.0%. Polar attention variant also limits 256K bits-per-byte likelihood degradation to 1.26x, with a 1.5-point mean penalty on downstream task evaluation. Raven instead leads at 256K length BABILong test and provide constant latency during decoding regardless of sequence length. Lastly, our independent environment audit finds a striking evidence that NoPE variant checkpoints with identical validation curve, significantly degrade 256K nats by …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2606.25156
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.