Abstract
This paper introduces the entropy gap—a novel information-theoretic metric—to explain the performance variations of optimizers like SGD and Adam across different neural architectures. While traditional metrics like Hessian sharpness focus on the speed of escaping saddle points, they fail to capture the uniformity of this escape. By bridging optimization dynamics with matrix entropy, the authors derive a new PAC-Bayes generalization bound that explicitly incorporates the entropy gap. Empirical results on CIFAR10 demonstrate that a lower entropy gap correlates with superior generalization, explaining why SGD excels on ResNets while Adam is preferred for Vision Transformers (ViTs).
Introduction
A persistent challenge in deep learning is the "mismatch" between optimizers and architectures. For instance, Stochastic Gradient Descent (SGD) remains the gold standard for ResNets, yet it often underperforms on Vision Transformers (ViTs), where adaptive methods like Adam or Lion are typically favored. Conventional explanations rely on sharpness (the trace of the Hessian matrix), suggesting that flatter minima generalize better. However, sharpness only describes the total magnitude of curvature, not its distribution. This paper argues that the uniformity of escape from saddle points is a critical, overlooked factor in stability and generalization.
Methods
The authors propose moving beyond simple trace statistics by utilizing Matrix Entropy to define the Entropy Gap.
The Entropy Gap
The entropy gap is defined as the difference between the maximum possible entropy ($\ln n$) and the actual von Neumann entropy of the Hessian matrix. A small gap indicates that curvature is distributed uniformly across all directions, whereas a large gap suggests the optimizer is biased toward specific directions, often associated with a high condition number and a "difficult" optimization landscape.
Theoretical Framework
The research establishes a formal link between Hessian uniformity and performance through a new PAC-Bayes generalization bound. This bound mathematically demonstrates that generalization error is constrained by a term involving the entropy gap $( \ln d - H(H) )$, providing a theoretical foundation for the importance of Hessian entropy.
Information-Theoretic Analysis of Lion
The authors also apply the Information Bottleneck (IB) principle to the Lion optimizer. By modeling Lion’s update rule as a binary classification problem, they show that its sign function acts as an information compressor. Using the maximum entropy principle, they derive that a tanh function serves as a theoretically superior "soft" version of Lion's sign function for compressing update information.
Experimental Results
The study evaluates these metrics using ResNet18 and ViT architectures trained on the CIFAR10 dataset.
Optimizer-Architecture Dynamics
The empirical results highlight a clear correlation between the entropy gap and final test accuracy. On ResNet18, SGD maintains a significantly lower entropy gap and achieves higher accuracy than Adam. Conversely, on ViTs, Adam demonstrates a lower entropy gap and superior performance compared to SGD.
As shown in the tables above, the "winning" optimizer for each architecture is the one that minimizes the entropy gap. For ResNet18, SGD's final entropy gap of 0.57 corresponds to a 95.21% accuracy, while Adam's higher gap of 1.48 results in a lower 93.53% accuracy. For ViTs, Adam's lower gap (0.32 vs 0.34) aligns with its higher accuracy (87.21% vs 83.23%).
Discussion and Conclusion
The entropy gap provides a more nuanced lens for understanding optimizer behavior than sharpness alone. It highlights that "uniformity is king"—stable training requires moving effectively in all directions, not just a few.
While theoretically robust, calculating the full Hessian entropy is computationally expensive for large-scale models. Future work should focus on efficient approximations for billion-parameter models and empirical validation of the proposed "Lion-Tanh" variant. Extending this analysis to emerging architectures like State Space Models could further generalize these information-theoretic insights.
Summary updated