Information-Theoretic Perspectives on Optimizers

Zhiquan Tan, Weiran HuangarXiv 2025 · DOI · arXiv

Abstract

This paper introduces the entropy gap—a novel information-theoretic metric—to explain the performance variations of optimizers like SGD and Adam across different neural architectures. While traditional metrics like Hessian sharpness focus on the speed of escaping saddle points, they fail to capture the uniformity of this escape. By bridging optimization dynamics with matrix entropy, the authors derive a new PAC-Bayes generalization bound that explicitly incorporates the entropy gap. Empirical results on CIFAR10 demonstrate that a lower entropy gap correlates with superior generalization, explaining why SGD excels on ResNets while Adam is preferred for Vision Transformers (ViTs).

Introduction

A persistent challenge in deep learning is the "mismatch" between optimizers and architectures. For instance, Stochastic Gradient Descent (SGD) remains the gold standard for ResNets, yet it often underperforms on Vision Transformers (ViTs), where adaptive methods like Adam or Lion are typically favored. Conventional explanations rely on sharpness (the trace of the Hessian matrix), suggesting that flatter minima generalize better. However, sharpness only describes the total magnitude of curvature, not its distribution. This paper argues that the uniformity of escape from saddle points is a critical, overlooked factor in stability and generalization.

Methods

The authors propose moving beyond simple trace statistics by utilizing Matrix Entropy to define the Entropy Gap.

The Entropy Gap

The entropy gap is defined as the difference between the maximum possible entropy ($\ln n$) and the actual von Neumann entropy of the Hessian matrix. A small gap indicates that curvature is distributed uniformly across all directions, whereas a large gap suggests the optimizer is biased toward specific directions, often associated with a high condition number and a "difficult" optimization landscape.

Theoretical Framework

The research establishes a formal link between Hessian uniformity and performance through a new PAC-Bayes generalization bound. This bound mathematically demonstrates that generalization error is constrained by a term involving the entropy gap $( \ln d - H(H) )$, providing a theoretical foundation for the importance of Hessian entropy.

Information-Theoretic Analysis of Lion

The authors also apply the Information Bottleneck (IB) principle to the Lion optimizer. By modeling Lion’s update rule as a binary classification problem, they show that its sign function acts as an information compressor. Using the maximum entropy principle, they derive that a tanh function serves as a theoretically superior "soft" version of Lion's sign function for compressing update information.

Experimental Results

The study evaluates these metrics using ResNet18 and ViT architectures trained on the CIFAR10 dataset.

Optimizer-Architecture Dynamics

The empirical results highlight a clear correlation between the entropy gap and final test accuracy. On ResNet18, SGD maintains a significantly lower entropy gap and achieves higher accuracy than Adam. Conversely, on ViTs, Adam demonstrates a lower entropy gap and superior performance compared to SGD.

Test accuracy across training steps for different configurations.
Entropy gap measurements across training steps.

As shown in the tables above, the "winning" optimizer for each architecture is the one that minimizes the entropy gap. For ResNet18, SGD's final entropy gap of 0.57 corresponds to a 95.21% accuracy, while Adam's higher gap of 1.48 results in a lower 93.53% accuracy. For ViTs, Adam's lower gap (0.32 vs 0.34) aligns with its higher accuracy (87.21% vs 83.23%).

Discussion and Conclusion

The entropy gap provides a more nuanced lens for understanding optimizer behavior than sharpness alone. It highlights that "uniformity is king"—stable training requires moving effectively in all directions, not just a few.

While theoretically robust, calculating the full Hessian entropy is computationally expensive for large-scale models. Future work should focus on efficient approximations for billion-parameter models and empirical validation of the proposed "Lion-Tanh" variant. Extending this analysis to emerging architectures like State Space Models could further generalize these information-theoretic insights.

Summary updated

Abstract

The interplay of optimizers and architectures in neural networks is complicated and hard to understand why some optimizers work better on some specific architectures. In this paper, we find that the traditionally used sharpness metric does not fully explain the intricate interplay and introduces information-theoretic metrics called entropy gap to better help analyze. It is found that both sharpness and entropy gap affect the performance, including the optimization dynamic and generalization. We further use information-theoretic tools to understand a recently proposed optimizer called Lion and find ways to improve it.
BibTeX
@misc{tan2025information,
  author = {Zhiquan Tan and Weiran Huang},
  title = {{Information-Theoretic Perspectives on Optimizers}},
  year = {2025},
  eprint = {2502.20763},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.20763},
}
Figure