Over-parameterization and the Risk of Training Data Memorization

50 concepts52 papers2 proposals

Modern deep learning relies on models with millions, or even billions, of parameters. While this high capacity allows neural networks to represent complex functions, it creates a fundamental tension: the model often possesses far more degrees of freedom than there are data points to constrain them. This state of over-parameterization frequently leads to overfitting, where a network memorizes the specific noise and coincidences of a training set rather than extracting generalizable patterns.

The Mechanism of Over-parameterization

When a model is over-parameterized, it can achieve "interpolation"—zero error on the training data—by essentially creating a lookup table of its inputs. Krogh et al. provided an early foundation for understanding this, demonstrating that without constraints, high-capacity networks choose the most complex mathematically valid solution rather than the simplest one. This is particularly acute in "fat data" regimes, such as genomics, where Romero et al. observed that having millions of genetic markers for only a few thousand patients makes standard networks highly prone to memorizing noise.

The risk isn't just limited to clean training data. Donhauser et al. found that interpolation is actively harmful to robust generalization. Even in noise-free environments, fitting the training data perfectly can compromise a model's stability when faced with adversarial attacks or distribution shifts. Similarly, Zhu identified "robust overfitting" in pre-trained language models like BERT, where the model eventually stops learning general linguistic features and begins memorizing specific adversarial perturbations.

Regularization as Information Filtering

To combat the tendency of over-parameterized models to memorize, researchers use regularization to force the network to be "simpler" or more efficient. These methods act as filters that suppress noise.

  • Stochastic Constraints: Techniques like Dropout and its variants introduce noise during training to prevent co-adaptation of neurons. Gao et al. proved that dropout's impact on model complexity is profound, especially as networks grow deeper, by effectively reducing the Rademacher complexity of the hypothesis space. Le further automated this by using reinforcement learning to find optimal, architecture-specific dropout patterns.
  • Bayesian Sparsity: Instead of manual tuning, Bayesian methods treat connectivity as a latent variable. Lin et al. introduced Adaptive Empirical Bayesian (AEB) methods to automatically induce sparsity, while Partaourides et al. developed DropConnect++ to infer which synaptic connections are redundant.
  • Information Bottlenecks: Some methods explicitly limit the information flow. Blot et al. proposed REVE to compress internal representations, retaining only the information necessary for prediction. Srivastava et al. applied Variational Information Bottleneck theory to human action recognition, achieving massive compression by stripping away non-generalizable variations in lighting and motion.

Structural Efficiency and Inductive Biases

Another way to mitigate over-parameterization is to change the architecture itself, replacing dense, flexible layers with structures that have fewer degrees of freedom.

Lin famously introduced "Network In Network," which replaced overfit-prone fully connected layers with Global Average Pooling. This structural change forces the model to learn more interpretable and generalizable spatial features. Similarly, Zhai proposed Doubly Convolutional Neural Networks, which use "meta-filters" to generate related filters, imposing a parameter-sharing constraint that acts as a powerful regularizer.

In the realm of sequence modeling, Ye et al. demonstrated that representing massive weight matrices as low-rank tensors (Block-Term Decomposition) could reduce parameters by over 17,000x while actually increasing accuracy. This suggests that much of the parameter count in standard models is not just redundant, but potentially detrimental.

The Benefits of Pruning and Diversity

Finally, reducing the number of parameters after or during training can serve as a defense mechanism. Jordão showed that pruning redundant filters is a potent defense against adversarial attacks, as it removes the excess capacity that attackers exploit. To ensure the remaining parameters are useful, Lin et al. developed techniques to minimize "hyperspherical energy," encouraging neurons to be as diverse as possible rather than redundant.

By understanding that over-parameterization is a double-edged sword—providing the flexibility to learn but also the capacity to memorize—researchers can better design constraints that steer models toward true generalization.

Key Foundational Work

A Simple Weight Decay Can Improve Generalization (1991) — contributed 1 concept elements to this direction.
Learning a Parametric Embedding by Preserving Local Structure (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_over_parameterization_8025bb62,
  title = {over-parameterization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/over-parameterization/rdr_8025bb62eb654071885ac5bb74f1c9b1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_over_parameterization_8025bb62_v1,
  title = {over-parameterization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/over-parameterization/rdr_8025bb62eb654071885ac5bb74f1c9b1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake