For years, deep learning practitioners have relied on a set of geometric "rules of thumb" to explain why certain models generalize better than others. The most famous of these is the "flat minima" hypothesis: the idea that if a model lands in a wide, shallow valley of the loss landscape, it will perform better on unseen data than if it lands in a narrow, sharp pit.
However, a growing body of research suggests that many of these geometric measures—sharpness, smoothness, and even fractal dimension—may not be the causal drivers of generalization we once thought. Instead, they often act as proxies or "bystanders" that merely reflect the hyperparameters used during training.
The Proxy Problem
The core challenge in deep learning theory is finding a "generalization measure"—a formula that can predict a model's test performance using only its training data and weights. While many proposed measures correlate well with performance across a standard suite of models, Gintare Karolina Dziugaite and colleagues argue in In search of robust measures of generalization that these correlations are often non-causal.
To prove a measure is truly capturing the "why" of generalization, it must survive a stress test of distributional robustness. If we intervene on the model—by changing its depth, the training set size, or the learning rate—the measure should still track performance. Instead, researchers are finding that most measures fail catastrophically when these underlying factors are shifted.
Sharpness as a Learning Rate Shadow
Consider the "flatness" of a minimum. In A Modern Look at the Relationship between Sharpness and Generalization, Maksym Andriushchenko and his co-authors demonstrate that for modern architectures like Transformers, sharpness is a poor predictor of success.
The intuition is simple: high learning rates tend to push models toward flatter regions of the loss landscape. Because high learning rates often (but not always) improve generalization, flatness appears to be the cause of the improvement. In reality, sharpness is often just a mathematical shadow cast by the optimization schedule. When the learning rate is held constant, the predictive power of sharpness frequently vanishes.
A similar "bystander effect" plagues more exotic geometric measures. Charlie B. Tan and Anthea Monod investigated the fractal dimension of optimization trajectories in On the Limitations of Fractal Dimension as a Measure of Generalization. They found that while fractal dimension correlates with the generalization gap, this link is largely a byproduct of batch size and learning rate choices. Once these hyperparameters are mathematically controlled, the fractal measure loses its ability to predict how well a model will generalize.
Decoupling Geometry from Performance
If the shape of the loss landscape were a reliable map, we would expect "smooth" paths to lead to better models. Yet, Jonathan Frankle and Tiffany J. Vlaar show in What Can Linear Interpolation of Neural Network Loss Landscapes Tell Us? that this relationship is easily decoupled.
By adjusting weight decay or simplifying data, one can create a perfectly "smooth" 1D linear path from initialization to the final solution—removing all visual "barriers" in the landscape—while simultaneously making the model's actual test accuracy worse. This suggests that the visual or geometric simplicity of the optimization path is not a reliable indicator of the difficulty of the learning task or the quality of the resulting model.
Toward Functional Measures
If geometry is a deceptive guide, where should we look instead? Some researchers are moving away from the shape of the weights and toward the functional behavior of the network.
In Robustness to Pruning Predicts Generalization in Deep Neural Networks, Clare Lyle and Yarin Gal propose prunability as a more robust metric. Prunability measures the smallest fraction of a network that can be kept without degrading training performance. Unlike static geometric measures, prunability tracks complex phenomena like "double descent" more accurately, suggesting that the internal redundancy of a network—how much of its capacity is actually "necessary"—might be a more fundamental driver of generalization than the curvature of the loss surface.
The shift in the field is clear: we are moving from observing correlations in the "geography" of neural networks to seeking the causal "mechanisms" of learning. Understanding generalization requires us to look past the proxies and ask what the optimization process is actually doing to the model's functional logic.