Classical statistical learning theory warns that models with more parameters than data points will overfit, creating highly complex, jagged decision boundaries to memorize noise. Yet, modern deep learning models routinely operate in this "overparameterized" regime and generalize exceptionally well. This phenomenon is known as double descent: as model capacity increases, test error first increases up to an "interpolation threshold" (where the model is just large enough to fit the training data perfectly), and then decreases as capacity continues to grow.
Why does generalization improve past the interpolation threshold? A growing body of research suggests that the answer lies in the smoothness of the learned function. In deep learning, smoothness is often formalized via the Lipschitz constant—a measure of how much a function's output can change in response to a small change in its input. While early work by Mihaela Rosca et al. in A case for new neural network smoothness constraints noted that smoothness is a crucial but poorly understood "secret sauce" for generalization, recent findings reveal a striking structural pattern: the smoothness of a neural network itself undergoes a double descent.
The Empirical Discovery of Lipschitz Double Descent
Intuitively, one might expect that adding more parameters gives a network the capacity to become more complex and "rougher." However, empirical studies show the exact opposite.
When tracking the empirical Lipschitz constant across networks of varying widths, researchers observed a non-monotonic trend that perfectly mirrors the double descent curve of test error. As demonstrated by Matteo Gamba et al. in Overparameterization Implicitly Regularizes Input-Space Smoothness and On the Lipschitz Constant of Deep Networks and Double Descent, the Lipschitz constant increases as model size grows toward the interpolation threshold, reaching a peak where the network is just barely able to fit the training data. Crucially, as the model becomes massively overparameterized, the Lipschitz constant decreases.
This finding is further supported by Anton Xue et al. in Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks and verified through rigorous lower bounds by Grigory Khromov in Some Fundamental Aspects about Lipschitz Continuity of Neural Networks. Grigory Khromov also notes that during training, the Lipschitz constant initially rises as the network extracts features, but the final converged model is significantly smoother when width is large. This non-monotonic behavior indicates that overparameterization acts as an implicit regularizer, driving the network toward smoother input-space mappings.
The Mechanism: Functional Smoothing vs. Structural Density
To understand why larger networks choose smoother functions, we must look at how their internal geometry changes. For instance, a network using ReLU activations splits the input space into a collection of distinct linear regions. A common assumption is that a network with more linear regions is inherently more complex.
However, Matteo Gamba et al. challenge this assumption in Are All Linear Regions Created Equal?. They show that while the number of linear regions (structural density) grows monotonically with model size, the effective nonlinearity (measured by absolute deviation) follows the double descent curve. Past the interpolation threshold, adjacent linear regions begin to align, sharing very similar slopes and biases.
This process, termed functional smoothing, means that even though a massive network has the structural capacity to be highly complex, standard optimization algorithms naturally guide it to use that capacity to "glue" adjacent regions together into a smooth, nearly linear mapping. As a result, as shown by Matteo Gamba et al. in Deep Double Descent via Smooth Interpolation, the network transitions from a "sharp" interpolation of data points to a "smooth" interpolation, where predictions are stable across large volumes of the input space.
Why Overparameterization is Necessary for Smoothness
Why can't a smaller model achieve the same level of smoothness while fitting the data? The core intuition is that fitting noisy data points while maintaining a low Lipschitz constant is a geometrically demanding task.
In A Law of Robustness for Two-Layers Neural Networks, Sebastien Bubeck et al. prove a fundamental trade-off: for a neural network to remain robust to small input changes while interpolating data, its size must be significantly larger than what is required for simple memorization. Specifically, they show that the Lipschitz constant of a shallow network scales inversely with the square root of the number of parameters relative to the dataset size. To achieve a constant level of smoothness, you need an overparameterization factor proportional to the number of data points.
Similarly, Hao Liu et al. demonstrate in Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint that approximating both a function's values and its derivatives (to ensure smoothness) requires vastly more network capacity than approximating values alone.
From an optimization perspective, expanding the parameter space alters the loss landscape. As argued by Andres R Masegosa et al. in PAC-Chernoff Bounds: Understanding Generalization in the Interpolation Regime, higher dimensionality allows gradient descent to locate interpolating solutions with higher loss concentration—solutions that remain smooth relative to the noise in the dataset. In contrast, a small network at the interpolation threshold is tightly constrained; it must twist and bend sharply to pass through every data point, resulting in a high Lipschitz constant and poor generalization.
Connecting Smoothness to Generalization Stability
The realization that overparameterization induces smoothness provides a bridge to classical learning theory. In Do We Really Need a New Theory to Understand the Double-Descent?, Luca Oneto et al. demonstrate that the well-established framework of Algorithmic Stability is fully capable of explaining double descent. When a model transitions into the overparameterized regime, the optimization process selects smoother functions, reducing the model's sensitivity to small perturbations in the training data. This stability directly translates to lower generalization error.
However, mathematically bounding this smoothness requires care. As noted by Ilja Kuzborskij et al. in Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping, traditional theoretical approaches that bound the Lipschitz constant using simple norm metrics (like the Frobenius norm of the weights) often scale poorly with network width, erroneously predicting that larger networks become rougher. The empirical reality of functional smoothing highlights the need for more sophisticated, data-dependent metrics to fully capture how overparameterization tames the complexity of deep networks.
Key Foundational Work
Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping (2021) — contributed 2 concept elements to this direction.
A case for new neural network smoothness constraints (2020) — contributed 1 concept elements to this direction.
A Law of Robustness for Two-Layers Neural Networks (2021) — contributed 1 concept elements to this direction.