R\'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2510.07758v3 Announce Type: replace Abstract: Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks. Unfortunately, the correlation between existing sharpness measures and generalization is not as strong as expected, and sometimes even contradiction occurs. To address this problem, a key observation in this paper is: what really matters for generalization is the average spread (or unevenness) of the spectrum of loss Hessian $\mathbf{H}$. For this reason, conventional sharpness measures, such as trace sharpness $\operatorname{tr}(\mathbf{H})$, which cares about the average value of the spectrum, or max-eigenvalue sharpness $\lambda_{\max}(\mathbf{H})$, which concerns the maximum spread of the spectrum, are not sufficient to well predict generalization. To characterize the average spread of the Hessian spectrum, we leverage the notion of R\'enyi entropy in information theory, which captures the unevenness of a probability vector and can thus be extended to a general non-negative vector, such as the Hessian spectrum at loss minima. Specifically, we propose R\'enyi sharpness, defined as the negative of the R\'enyi entropy of loss Hessian $\mathbf{H}$. Extensive experiments demonstrate that R\'enyi sharpness exhibits strong and consistent correlation with generalization in various scenarios. Moreover, two generalization bounds with respect to R\'enyi sharpness are established by exploiting its desirable reparametrization invariance property. Finally, as an initial attempt to exploit R\'enyi sharpness for …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2510.07758
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.