Generalization Bounds via Distance from Initialization

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 235 papers52 conceptsfrozen 2026-10-04

In classical statistical learning theory, overparameterization is usually a recipe for disaster. If a model has more parameters than data points, it should—in theory—be able to memorize the training set perfectly while failing to generalize to new data. Yet, modern deep learning thrives in this exact regime.

The resolution to this paradox lies in the geometry of the optimization path. Recent research suggests that neural networks generalize not because they have low capacity, but because the training process is "lazy": the weights stay remarkably close to their random initialization. By measuring the distance weights travel rather than their absolute magnitude, researchers have developed a new framework for understanding why massive models don't overfit.

The Failure of Absolute Norms

Traditional generalization bounds often rely on the absolute norms of weight matrices (e.g., the Frobenius norm $\|W\|$). However, as Amit Daniely and Elad Granot point out in On the Sample Complexity of Two-Layer Networks, these bounds become "vacuous" or uninformative for modern networks. Because weights are initialized randomly, their absolute norm is already large at the start of training.

Instead, a more predictive metric is the initialization-relative weight norm: $\|W - W_0\|$, where $W_0$ is the weight matrix at initialization. Behnam Neyshabur and colleagues demonstrated in Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks that this distance is a far more accurate predictor of generalization than the raw number of parameters or the absolute weight magnitudes. In highly overparameterized regimes, weights move very little to reach a solution, effectively restricting the model to a "simple" hypothesis space near the starting point.

The Geometry of Signal and Noise

Why does staying near initialization prevent overfitting? The intuition is that structured "signal" in the data is easier to find than random "noise."

Mahdi Soltanolkotabi, Mingchen Li, and Samet Oymak proved in Gradient Descent with Early Stopping is Provably Robust to Label Noise that fitting random noise requires the weights to travel a much larger distance from initialization than fitting structured data. This creates a natural separation in the loss landscape:

  1. Generalizing solutions are "near" the initialization point.
  2. Overfitting solutions (which memorize noise) exist, but they are "far" away.

This geometric reality is confirmed by Daniil Merkulov and Ivan Oseledets in their Empirical study of extreme overfitting points. They found that while it is possible to find weights that yield 100% training accuracy and 0% test accuracy, these "catastrophic minima" are located significantly further from standard initializations than the solutions found by standard gradient descent.

Size-Free Complexity

If the distance from initialization is small, the effective complexity of the model becomes independent of its actual width. Hanie Sedghi and Philip M. Long leveraged this in Size-free generalization bounds for convolutional neural networks, showing that CNN generalization can be bounded by the distance weights travel, regardless of the number of pixels or internal dimensions.

This concept is formalized through Approximate Description Length (ADL). In Generalization Bounds for Neural Networks via Approximate Description Length, Daniely and Granot argue that we should measure how many bits are needed to "sketch" the change in the network's function. Because the weights stay close to their initial values, the "sketch" of the training progress is much smaller than the network itself, leading to sample complexities that are sub-linear in the number of parameters.

Implicit Regularization via Optimization

The distance from initialization acts as an implicit regularizer. Even without explicit penalties like weight decay, the optimization algorithm (Gradient Descent or SGD) naturally prefers the nearest viable solution.

Hossein Taheri and Christos Thrampoulidis show in Sharper Guarantees for Learning Neural Network Classifiers that the optimization path itself limits the effective complexity of the model. This is why "lazy training"—where weights move minimally—is not a bug, but a feature that ensures generalization. Similarly, in the context of fine-tuning, Haotian Ju and others have used PAC-Bayes theory to show that maintaining a small distance between pre-trained and fine-tuned weights is the primary mechanism for preventing overfitting on small target datasets.

In summary, modern neural networks generalize because they are "close-to-initialization" learners. The vast parameter space is not a liability but a vast sea in which the optimization algorithm quickly finds a nearby, simple island of signal before it has the chance to wander off into the distant, complex reefs of noise.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 29 papers
  2. Version 1 · 2026-10-04 · 35 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_initialization_relative_weight_distance_908a2002,
  title = {initialization-relative weight distance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initialization-relative-weight-distance/rdr_908a200296d74277aaf86abe64103dc4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalization_bounds_via_distance_from__908a2002_v1,
  title = {Generalization Bounds via Distance from Initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-bounds-via-distance-from-initialization/rdr_908a200296d74277aaf86abe64103dc4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}