Classical learning theory presents a fundamental paradox when applied to deep neural networks. According to traditional statistical wisdom, a model with significantly more parameters than data points should suffer from massive overfitting, memorizing noise rather than extracting signal. Yet, modern deep learning routinely relies on overparameterized networks that generalize remarkably well to unseen data.
To resolve this paradox, researchers have shifted their focus from counting raw parameters to measuring the geometric trajectory of optimization. The core insight is simple yet profound: overparameterized models generalize well because their weights remain close to their initial values throughout training. By measuring model complexity as the distance from initialization rather than the distance from zero, we can derive tight, non-vacuous generalization bounds that accurately reflect empirical reality.
The Geometry of Lazy Training
Why do massive neural networks stay close to their starting points? In a highly overparameterized regime, the network operates in what is frequently called the "lazy training" or neural tangent kernel regime. Because the network has an immense capacity, it does not need to radically alter its parameters to fit the training data. Instead, each individual weight only needs to shift by a microscopic amount to collectively achieve a global minimum.
Historically, norm-based generalization bounds measured the absolute magnitude of the weight matrices—effectively calculating their distance from the origin ($0$). However, random initializations start far from the origin. Measuring complexity this way yields loose, uninformative bounds that scale poorly with network width.
To address this, Behnam Neyshabur et al. demonstrated in The role of over-parametrization in generalization of neural networks and Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks that the Euclidean distance between trained weights and their random initialization points serves as a far more accurate predictor of generalization capacity. Because weights stay anchored near their starting values, initialization acts as an implicit regularizer that constrains the effective search space.
Traditional Norm-Based View:
Origin (0) -----------------------------> Final Weights (W) [Massive Absolute Norm]
Initialization-Relative View:
Origin (0) ---------------> Init (W_0) -> Final Weights (W) [Small Travel Distance]
Building on this geometric perspective, Amit Daniely et al. introduced the Approximate Description Length (ADL) framework in Generalization Bounds for Neural Networks via Approximate Description Length. By calculating how many bits are required to "sketch" a network's function relative to its initial state, the authors proved that sample complexity can be sub-linear in the number of parameters. Furthermore, in On the Sample Complexity of Two-Layer Networks: Lipschitz Vs. Element-Wise Lipschitz Activation, the authors showed that when weight updates are constrained relative to initialization, the required sample complexity depends only logarithmically on the network's width, explaining why massive models do not overfit.
This perspective has been extended across various architectures to ensure theoretical bounds remain tight at initialization:
- Convolutional Networks: Antoine Ledent et al. in Norm-Based Generalisation Bounds for Deep Multi-Class Convolutional Neural Networks and Philip M. Long et al. in Generalization bounds for deep convolutional neural networks developed bounds based on weight travel distance that scale gracefully with depth and avoid exponential dependencies.
- Residual & Implicit Models: Florian Graf et al. utilized group norm distances from initialization to quantify excess capacity in On Measuring Excess Capacity in Neural Networks, while Hongyang Gao et al. proved initialization-sensitive bounds for infinite-depth implicit models in On the optimization and generalization of overparameterized implicit neural networks.
- Optimization Dynamics: Xiaoge Deng et al. demonstrated in Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness that the distance between initial parameters and the global optimum directly dictates the stability-based error bound for distributed optimization.
Geometric Separation of Signal and Noise
Measuring the distance from initialization does more than just provide tighter math; it reveals a fundamental mechanism for how networks distinguish true patterns from random noise.
In Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks, Mahdi Soltanolkotabi et al. analyzed the weight displacement metric during gradient descent. They discovered a geometric separation in how networks learn: fitting a coherent, underlying signal requires very little parameter movement from the initial state. Conversely, memorizing corrupted or noisy labels requires the network to force its weights along highly complex trajectories, moving significantly further from initialization.
By employing early stopping, practitioners keep the model within a tight geometric radius around $W_0$, naturally capturing the clean signal while remaining too close to initialization to fit the noise.
[Init W_0]
/ \
/ \ (Longer distance required
/ \ to memorize noise)
v v
[Clean Signal] [Noisy Labels]
This relationship between weight travel and generalization is also visible in sparse networks. Zheng He et al. explored this in Sparse Double Descent: Where Network Pruning Aggravates Overfitting, showing that at moderate sparsity levels, networks often exhibit a "sparse double descent" where they overfit severely. The authors found that the total Euclidean distance weights travel from initialization is a more reliable predictor of this overfitting than traditional loss landscape flatness metrics; models that move further from their starting configuration are consistently more prone to memorizing noise.
The Scale of Initialization and Training Trajectories
The choice of initialization scale also dictates the qualitative nature of the features a network can learn. When models are initialized with extremely small weights, they follow a distinct trajectory.
As shown by Hanxu Zhou et al. in Towards Understanding the Condensation of Neural Networks at Initial Training, small weight initializations cause hidden neurons to initially "condense" into a limited number of orientations before expanding to learn high-dimensional features. This transition from a simple, low-dimensional alignment to complex feature learning is critical for generalization.
Similarly, Loek van Rossem et al. found in Algorithm Development in Neural Networks: Insights from the Streaming Parity Task that small initial weights are required for recurrent networks to discover robust underlying algorithms; large initial weights cause the model to satisfy objectives via memorization instead of representation merging.
This initialization distance also explains training delays like "grokking"—where a model suddenly generalizes long after overfitting the training data. In A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation, Etienne Boursier et al. proved that larger initial weights place the model further from the optimal-norm solution, requiring a prolonged period of "Riemannian drift" along the solution manifold before the model's complexity is sufficiently reduced to generalize.
Anchoring Fine-Tuning to Pre-trained Representations
The principles of initialization-relative distance extend naturally from random initializations to pre-trained checkpoints. In the context of transfer learning and fine-tuning, the pre-trained weights ($W_{pre}$) serve as a highly informed starting state that already encodes rich, generalizable features.
When fine-tuning on a small downstream dataset, the risk of overfitting or "catastrophic forgetting" is high. To combat this, Henry Gouk et al. proved in Distance-Based Regularisation of Deep Networks for Fine-Tuning that generalization bounds during fine-tuning depend on the distance weights travel from their pre-trained starting point rather than the total parameter count.
By explicitly penalizing the $L_2$ or Frobenius distance between the evolving weights and the pre-trained initialization, algorithms can guarantee that the model retains robust source features. For instance:
- Dongyue Li et al. introduced distance-based penalties in Improved Regularization and Robustness for Fine-tuning in Neural Networks to bound generalization error and handle noisy downstream labels.
- Li Zhang 0084 et al. leveraged PAC-Bayes generalization theory in Distance-Based Weight Transfer for Fine-Tuning From Near-Field to Far-Field Speaker Verification to theoretically justify weight transfer regularization, showing it yields a tighter bound and reduces overfitting.
While pure distance metrics provide a strong foundation, they are occasionally refined to capture finer geometric nuances. As noted by Haotian Ju et al. in Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees, simple distance-based measures can sometimes fail to predict exact test performance if the loss landscape is highly asymmetric, prompting the use of second-order Hessian constraints alongside distance boundaries to ensure robust adaptation.
Ultimately, whether considering a model starting from a random distribution or a massive pre-trained checkpoint, measuring the distance from initialization provides the mathematical and intuitive bridge explaining why overparameterized models succeed: they find simple, robust solutions right in their own neighborhood.
Key Foundational Work
Generalization Bounds for Neural Networks via Approximate Description Length (2019) — contributed 3 concept elements to this direction.
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks (2018) — contributed 2 concept elements to this direction.
The role of over-parametrization in generalization of neural networks (2019) — contributed 2 concept elements to this direction.