Abstract
Optimization theory relies on "clean" assumptions like convexity and smoothness, but deep learning is famously messy. This paper introduces empirical metrics to show that while global assumptions fail, the underlying mathematical identities used in convex analysis often hold surprisingly well, whereas standard smoothness-based "descent" logic frequently contradicts what actually happens during training. By measuring these low-level identities, we can identify which theoretical frameworks explain modern AI performance and which are merely convenient fictions.
The Theory-Practice Gap
There is a massive "theory-practice gap" in deep learning. Researchers develop optimization algorithms (like Adam or SGD) and prove they work using specific mathematical assumptions. However, deep neural networks are non-convex and non-smooth due to components like ReLU activations. Instead of asking "is the function convex?"—to which the answer is almost always no—this research asks if the specific algebraic identities used in proofs hold true during real training runs.
Methodology
The authors propose several "on-the-fly" empirical metrics to monitor training across a broad range of tasks, including ResNet-18 on ImageNet/CIFAR-10 and BERT/GPT-2 pre-training:
- Convexity Ratio: Measures the ratio between actual loss reduction and the reduction predicted by a linear approximation. A ratio $>0$ suggests convex-style analysis might still be valid.
- Instantaneous Smoothness: A computationally efficient proxy for the Hessian's spectral norm, measuring how fast the gradient changes between consecutive steps.
- Update Correlation: Measures the alignment between the gradient and the optimizer's step. Standard theory assumes this is negative (moving "downhill").
- Random Scaling (RS): A technique from non-smooth optimization theory used to verify if non-smooth frameworks better match reality.
Key Insights and Ideas
The "Convexity Rescue"
Surprisingly, even though deep learning is non-convex, the Convexity Ratio is typically positive. This explains why optimizers derived from Online Convex Optimization (like AdaGrad and Adam) perform so well—the landscape is "sufficiently" convex along the path the optimizer actually travels.
Smoothness is a Myth
The "smoothness" constants measured in practice are enormous ($10^3$ to $10^6$). The objective is neither globally nor locally smooth in the way traditional theory requires. The authors observed that smoothness/sharpness increases early in training and then stabilizes, confirming the "edge of stability" phenomenon even in large-scale models like GPT-2 and BERT.
The Bouncing Optimizer
In most deep learning tasks, the Update Correlation is actually positive on average. This contradicts the basic "descent lemma" used in almost all smooth optimization proofs. It suggests the optimizer isn't sliding down a smooth hill but is instead "bouncing" back and forth between the walls of a narrow ravine.
Key Results
- Convexity Ratio: On ImageNet, the ratio is consistently $>1$ (indicating convexity). On CIFAR-10, it stays between $0.5$ and $1.0$, suggesting "weak quasi-convexity."
- Non-Smooth Theory Wins: By applying Random Scaling (RS), the authors forced the update correlation to become negative, aligning experimental results with the predictions of non-smooth optimization theory. This suggests that non-smooth analysis might be a more "honest" way to model deep learning than smooth analysis.
- Efficiency: The new smoothness metric tracks actual Hessian sharpness almost perfectly while being significantly cheaper to calculate, providing a new tool for monitoring model stability.
Limitations and Future Work
While this study covers industry standards like SGDM and AdamW, emerging optimizers like Lion or Shampoo remain untested. Additionally, the practice of resetting non-instantaneous metrics at the start of each epoch might obscure long-term trends. The next step for the field is to leverage these findings—specifically the positive convexity ratio and positive update correlation—to design new optimizers specifically built for these "messy" empirical realities.
Summary updated