Modern neural networks often possess millions or billions of parameters, yet they generalize to unseen data with surprising efficiency. This behavior contradicts classical statistical learning theory, which suggests that such massive "capacity" should lead to catastrophic overfitting. The resolution to this paradox lies in the realization that Stochastic Gradient Descent (SGD) does not explore the entire high-dimensional parameter space. Instead, the optimization process restricts the model to a much smaller, highly structured subset of the weight space.
Researchers have recently begun to quantify this structure using fractal geometry, showing that the "effective complexity" of a model is determined by the fractal dimension of its optimization trajectory rather than its raw parameter count.
The Fractal Nature of SGD
When we train a neural network, the optimizer (typically a variant of SGD) moves through the weight space in a noisy, iterative fashion. Alexander Camuto, George Deligiannidis, and Murat A. Erdogdu demonstrated in Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms that this path is not random noise. Instead, the algorithm settles into a stationary distribution that exhibits fractal-like properties.
In this view, the "intrinsic complexity" of a trained model is captured by the Hausdorff dimension of the regions visited by the optimizer. Even if a model has a million parameters (a $10^6$-dimensional space), the optimizer might only traverse a fractal set with a dimension of, say, 15. This lower "effective dimensionality" explains why the model behaves as if it has a much smaller capacity.
Measuring Effective Complexity
To turn this intuition into a rigorous tool, we need ways to measure the "roughness" or "complexity" of the path taken during training. Several approaches have emerged:
- Box-Counting Dimension: Benjamin Dupuis, George Deligiannidis, and Umut Simsekli utilize the upper box-counting dimension to measure the hypothesis space explored during training. In Generalization Bounds using Data-Dependent Fractal Dimensions, they show that this metric captures the effective degrees of freedom relative to the loss surface.
- Fractional Brownian Motion: Chengli Tan, Jiangshe Zhang, and Junmin Liu model the SGD trajectory as a fractal path driven by fractional Brownian motion. In Learning Non-Vacuous Generalization Bounds from Optimization, they use this to measure the "roughness" of the optimization path, providing a more accurate representation of complexity than parameter counts.
- Metric Space Magnitude: Taking a perspective from category theory, Rayna Andreeva and Bastian Rieck treat the weights as a "point cloud." In Metric Space Magnitude and Generalisation in Neural Networks, they use the concept of "magnitude" to identify the intrinsic dimension of the space occupied by these weights.
Achieving Non-Vacuous Bounds
The ultimate goal of this research is to produce non-vacuous generalization bounds. Historically, theoretical bounds for neural networks were often "vacuous"—meaning the predicted error was mathematically guaranteed to be less than 100%, which is trivial and useless for practitioners.
By replacing the total number of parameters with the fractal dimension of the trajectory, researchers are finally producing bounds that track the actual performance of large-scale models like ResNets and Vision Transformers. As Liyuan Liu and colleagues argue in Trajectory-Dependent Generalization Bounds for Pairwise Learning with φ-mixing Samples, the generalization performance is determined by the structured subset of the parameter space the optimizer actually navigates.
Removing Theoretical Constraints
Early iterations of fractal-based theory required the loss function to be "smooth" (specifically, Lipschitz continuous). However, modern deep learning often uses non-smooth components like ReLU activations.
In their 2023 work, Benjamin Dupuis and Umut Simsekli successfully removed these restrictive assumptions. Their approach in Generalization Bounds using Data-Dependent Fractal Dimensions makes fractal complexity analysis applicable to a much wider range of modern architectures, bridging the gap between abstract geometry and practical deep learning.
By focusing on the geometry of the path rather than the size of the container, this research direction provides a robust explanation for why bigger models don't necessarily mean worse generalization: they simply move through their vast parameter spaces with more intricate, low-dimensional precision.