Introduction
Most deep learning optimizers are designed with "bowls" (convexity) in mind, but neural network loss landscapes are actually full of "saddles"—regions characterized by negative curvature. This paper demonstrates that in these downward-sloping saddle directions, we should be taking much larger steps than standard optimization theory suggests. Doing so could significantly accelerate training and help models escape saddle points more efficiently.
The Problem with Convex Thinking
In the world of mathematical optimization, "convex" is king. If a function is convex, its landscape looks like a simple bowl, meaning any local minimum is guaranteed to be the global minimum. However, Deep Neural Networks (DNNs) are notoriously non-convex. Their loss landscapes are highly complex and littered with saddle points—regions where the surface curves upwards in some directions but downwards in others.
The "curvature" of this landscape is described by the Hessian matrix, which is the matrix of second-order partial derivatives. In a perfectly convex bowl, all of the Hessian's eigenvalues are positive. At a saddle point, however, some eigenvalues are negative, indicating downward slopes. Current popular optimizers like Adam or RMSProp were largely inspired by convex optimization logic. The authors argue that because we lack an efficient, principled way to manage these negative eigenvalues, we are potentially leaving significant training speedups on the table.
Experimental Methods
To study this phenomenon, the authors looked under the hood of a classic LeNet architecture (using ReLU activations) trained on the MNIST dataset. Because the Hessian matrix for even a small model is gargantuan—storing it for a modest network would require terabytes of RAM—they employed a few clever mathematical and computational tricks:
- Jacobian Vector Product (JVP) Trick: Instead of calculating and storing the entire Hessian matrix, they used a mathematical shortcut to compute the product of the Hessian and a vector. This allowed them to use Scipy's
eigshfunction to extract just the most extreme eigenvalues (representing the "sharpest" up-curves and "steepest" down-curves) without materializing the full matrix. - Empirical Line Search: For specific directions in the parameter space (eigenvectors), they manually tested a wide range of step sizes ($\alpha$) to observe which one actually reduced the loss the most in practice.
- Subsampling: They discovered that they could accurately estimate the Hessian's properties using only 5% of the training data. This subsampling made the heavy computations feasible without losing significant fidelity.
Key Insights
- Negative Curvature is a Gold Mine: The authors found that moving along directions of negative curvature (the "downward" parts of a saddle) often yields much larger drops in loss than moving along directions of positive curvature.
- Throw Out the Convex Rulebook: In convex optimization, the "optimal" step size is often inversely proportional to the curvature, specifically $1/\lambda$ (where $\lambda$ is the eigenvalue). While this rule held true for positive curvature in their experiments, it completely failed for negative curvature. In those downward directions, the optimal step size was much larger than $1/|\lambda|$.
- Early Stability: Surprisingly, the most extreme eigenvalues (both positive and negative) stabilize very quickly during training—often within the first epoch. Even as the total loss continues to plummet, the fundamental "shape" of the most extreme curvatures remains relatively constant.
- Locality: Negative curvature is strictly a local phenomenon. Because the loss function is bounded below (it cannot drop to negative infinity), any downward slope must eventually flatten out or curve back upwards.
Key Results
The empirical findings strongly support the theoretical intuition that negative eigenvalues behave fundamentally differently than positive ones.
Optimization Potential: As illustrated in the figure above, the potential loss decrease is consistently higher on the "left side" (negative eigenvalues) than the "right side" (positive eigenvalues). The blue dots representing negative eigenvalues achieve much deeper loss minimization, proving that these saddle directions are highly valuable for optimization.
The Negative Divergence: When comparing the empirical optimal step size ($\alpha^*$) to the theoretical $1/\lambda$, a stark contrast emerges. For positive eigenvalues (the right side of the plot), the data points closely follow the theoretical diagonal trend. However, for negative eigenvalues (the left side), the points deviate sharply from the $1/\lambda$ expectation. This suggests that standard second-order methods, like Newton's method, would take steps that are far too small in these negatively curved regions.
Limitations and Future Work
The authors are transparent about the current scope and limitations of their work:
- Scale: The experiments were limited to smaller models like LeNet on MNIST and CIFAR-10. It remains to be seen if these specific curvature patterns hold identically for massive architectures like ResNets or Transformers.
- Computational Overhead: Extracting eigenvalues at runtime is currently too slow to be a standard part of a regular training loop.
- The "New Optimizer" Tease: In the appendix, the authors propose a theoretical optimization method that maintains a running estimate of the most negative eigenvector to guide parameter updates. However, they note that they have not yet benchmarked this method in practice. This leaves a clear, exciting opening for future researchers to turn these observations into a functional, high-speed optimizer tailored specifically for the non-convex saddles of deep learning.
Summary updated