Generalization Bounds via Distance from Initialization

39 concepts28 papers1 proposal

Classical learning theory presents a fundamental paradox when applied to deep neural networks. According to traditional statistical wisdom, a model with significantly more parameters than data points should suffer from massive overfitting, memorizing noise rather than extracting signal. Yet, modern deep learning routinely relies on overparameterized networks that generalize remarkably well to unseen data.

To resolve this paradox, researchers have shifted their focus from counting raw parameters to measuring the geometric trajectory of optimization. The core insight is simple yet profound: overparameterized models generalize well because their weights remain close to their initial values throughout training. By measuring model complexity as the distance from initialization rather than the distance from zero, we can derive tight, non-vacuous generalization bounds that accurately reflect empirical reality.

The Geometry of Lazy Training

Why do massive neural networks stay close to their starting points? In a highly overparameterized regime, the network operates in what is frequently called the "lazy training" or neural tangent kernel regime. Because the network has an immense capacity, it does not need to radically alter its parameters to fit the training data. Instead, each individual weight only needs to shift by a microscopic amount to collectively achieve a global minimum.

Historically, norm-based generalization bounds measured the absolute magnitude of the weight matrices—effectively calculating their distance from the origin ($0$). However, random initializations start far from the origin. Measuring complexity this way yields loose, uninformative bounds that scale poorly with network width.

To address this, Behnam Neyshabur et al. demonstrated in The role of over-parametrization in generalization of neural networks and Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks that the Euclidean distance between trained weights and their random initialization points serves as a far more accurate predictor of generalization capacity. Because weights stay anchored near their starting values, initialization acts as an implicit regularizer that constrains the effective search space.

Traditional Norm-Based View:
Origin (0) -----------------------------> Final Weights (W)   [Massive Absolute Norm]

Initialization-Relative View:
Origin (0) ---------------> Init (W_0) -> Final Weights (W)   [Small Travel Distance]

Building on this geometric perspective, Amit Daniely et al. introduced the Approximate Description Length (ADL) framework in Generalization Bounds for Neural Networks via Approximate Description Length. By calculating how many bits are required to "sketch" a network's function relative to its initial state, the authors proved that sample complexity can be sub-linear in the number of parameters. Furthermore, in On the Sample Complexity of Two-Layer Networks: Lipschitz Vs. Element-Wise Lipschitz Activation, the authors showed that when weight updates are constrained relative to initialization, the required sample complexity depends only logarithmically on the network's width, explaining why massive models do not overfit.

This perspective has been extended across various architectures to ensure theoretical bounds remain tight at initialization:

Geometric Separation of Signal and Noise

Measuring the distance from initialization does more than just provide tighter math; it reveals a fundamental mechanism for how networks distinguish true patterns from random noise.

In Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks, Mahdi Soltanolkotabi et al. analyzed the weight displacement metric during gradient descent. They discovered a geometric separation in how networks learn: fitting a coherent, underlying signal requires very little parameter movement from the initial state. Conversely, memorizing corrupted or noisy labels requires the network to force its weights along highly complex trajectories, moving significantly further from initialization.

By employing early stopping, practitioners keep the model within a tight geometric radius around $W_0$, naturally capturing the clean signal while remaining too close to initialization to fit the noise.

         [Init W_0]
           /     \
          /       \  (Longer distance required
         /         \  to memorize noise)
        v           v
 [Clean Signal]   [Noisy Labels]

This relationship between weight travel and generalization is also visible in sparse networks. Zheng He et al. explored this in Sparse Double Descent: Where Network Pruning Aggravates Overfitting, showing that at moderate sparsity levels, networks often exhibit a "sparse double descent" where they overfit severely. The authors found that the total Euclidean distance weights travel from initialization is a more reliable predictor of this overfitting than traditional loss landscape flatness metrics; models that move further from their starting configuration are consistently more prone to memorizing noise.

The Scale of Initialization and Training Trajectories

The choice of initialization scale also dictates the qualitative nature of the features a network can learn. When models are initialized with extremely small weights, they follow a distinct trajectory.

As shown by Hanxu Zhou et al. in Towards Understanding the Condensation of Neural Networks at Initial Training, small weight initializations cause hidden neurons to initially "condense" into a limited number of orientations before expanding to learn high-dimensional features. This transition from a simple, low-dimensional alignment to complex feature learning is critical for generalization.

Similarly, Loek van Rossem et al. found in Algorithm Development in Neural Networks: Insights from the Streaming Parity Task that small initial weights are required for recurrent networks to discover robust underlying algorithms; large initial weights cause the model to satisfy objectives via memorization instead of representation merging.

This initialization distance also explains training delays like "grokking"—where a model suddenly generalizes long after overfitting the training data. In A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation, Etienne Boursier et al. proved that larger initial weights place the model further from the optimal-norm solution, requiring a prolonged period of "Riemannian drift" along the solution manifold before the model's complexity is sufficiently reduced to generalize.

Anchoring Fine-Tuning to Pre-trained Representations

The principles of initialization-relative distance extend naturally from random initializations to pre-trained checkpoints. In the context of transfer learning and fine-tuning, the pre-trained weights ($W_{pre}$) serve as a highly informed starting state that already encodes rich, generalizable features.

When fine-tuning on a small downstream dataset, the risk of overfitting or "catastrophic forgetting" is high. To combat this, Henry Gouk et al. proved in Distance-Based Regularisation of Deep Networks for Fine-Tuning that generalization bounds during fine-tuning depend on the distance weights travel from their pre-trained starting point rather than the total parameter count.

By explicitly penalizing the $L_2$ or Frobenius distance between the evolving weights and the pre-trained initialization, algorithms can guarantee that the model retains robust source features. For instance:

While pure distance metrics provide a strong foundation, they are occasionally refined to capture finer geometric nuances. As noted by Haotian Ju et al. in Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees, simple distance-based measures can sometimes fail to predict exact test performance if the loss landscape is highly asymmetric, prompting the use of second-order Hessian constraints alongside distance boundaries to ensure robust adaptation.

Ultimately, whether considering a model starting from a random distribution or a massive pre-trained checkpoint, measuring the distance from initialization provides the mathematical and intuitive bridge explaining why overparameterized models succeed: they find simple, robust solutions right in their own neighborhood.

Key Foundational Work

Generalization Bounds for Neural Networks via Approximate Description Length (2019) — contributed 3 concept elements to this direction.
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks (2018) — contributed 2 concept elements to this direction.
The role of over-parametrization in generalization of neural networks (2019) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 29 papers
  2. Version 1 · 2026-10-04 · 35 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_initialization_relative_weight_distance_908a2002,
  title = {initialization-relative weight distance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initialization-relative-weight-distance/rdr_908a200296d74277aaf86abe64103dc4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_initialization_relative_weight_distance_908a2002_v2,
  title = {initialization-relative weight distance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initialization-relative-weight-distance/rdr_908a200296d74277aaf86abe64103dc4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake