Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2607.03998v4 Announce Type: replace Abstract: The local sharpness of the loss, the top Hessian eigenvalue $\lambda_1$, determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector products. A single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step $\alpha$ brackets the directional curvature along the probed direction within the multiplicative band set by the backtracking factor: exactly the curvature averaged over the tested step, and empirically $q = g^\top H g/\|g\|^2$ to within that band. Across CIFAR-10, Fashion-MNIST and Imagenette, $\log\alpha$ tracks $\log\lambda_1$ at Pearson $-0.91$ to $-0.95$, and the relation survives a per-run detrending check at $-0.60$ to $-0.70$, a low-cost online Edge-of-Stability reading of the slow sharpness component. Used as a safeguard rather than a faster optimiser, the reading caps a too-large initial learning rate. A single fixed protocol, probing along Adam's own update direction at initialisation and over the first fifty optimiser steps and capping the rate at twice the smallest reading, removes every divergence across learning-rate grids spanning $10^{-3}$ to $3.0$ and at GPT-2 pretraining scale, and all but one marginal case across the further architectures we test, at about $1\%$ overhead, and it leaves training bit-identical whenever the cap does not bind. No constant in the protocol is tuned per architecture; this is the sense in which the safeguard is calibration-free. The guarantee is divergence, not …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2607.03998
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.