Generalization Theory for Shallow and Deep Neural Networks

This direction is split into other directions: shallow network theoretical constraints. Its last version stays readable and citable here.

Version 1 of 1364 papers369 conceptsfrozen 2026-10-04

In classical statistics, over-parameterization is a recipe for disaster. If a model has more parameters than data points, it should theoretically memorize the noise and fail to generalize. Yet, modern deep learning thrives in this regime. To understand why, researchers have built a rigorous "laboratory" around two-layer (shallow) neural networks, uncovering mechanisms like benign overfitting and implicit bias. The central challenge of the field today is determining whether these shallow insights scale to the deep, hierarchical architectures used in practice.

The Two-Layer Laboratory

Theoretical progress in neural networks often begins with the "one-hidden-layer" or "two-layer" architecture. This simplification is not just for convenience; it allows researchers to isolate the fundamental dynamics of optimization and generalization.

For instance, Alon Brutzkus and Amir Globerson proved in SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data that Stochastic Gradient Descent (SGD) possesses an inherent "inductive bias." Even when a network is wide enough to memorize anything, SGD naturally steers it toward simple, high-quality solutions. Similarly, Atsushi Nitanda, Taiji Suzuki, and Geoffrey Chinot showed that by replacing restrictive kernel assumptions with "separability" conditions, we can prove global convergence for networks with far less over-parameterization than previously thought necessary in Refined Generalization Analysis of Gradient Descent for Over-parameterized Two-layer Neural Networks with Smooth Activations on Classification Problems.

The Mechanism of Benign Overfitting

One of the most counterintuitive findings in this area is benign overfitting: the phenomenon where a model perfectly fits noisy training data (zero training error) but still performs well on test data.

In Benign Overfitting in Two-layer ReLU Convolutional Neural Networks, Yiwen Kou, Quanquan Gu, and colleagues identified a sharp "phase transition" in shallow CNNs. They found that whether overfitting is benign or harmful depends on the signal-to-noise ratio. This was further refined by Xuran Meng and Yuan Cao in Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Data, proving that these networks can achieve near-optimal accuracy even on non-linear XOR problems with label noise.

The intuition here is that over-parameterized networks learn in a low-dimensional "information space." As Samet Oymak and Zalan Fabian argue in Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian, the network's Jacobian matrix is often low-rank. This means the model captures meaningful patterns quickly while relegating noise to a "nuisance space" where learning is much slower, effectively allowing early stopping to act as a regularizer.

Feature Learning and Contamination

Beyond just fitting curves, neural networks are feature manipulators. Ruoqi Shen, Sebastien Bubeck, and Suriya Gunasekar proposed in Data Augmentation as Feature Manipulation that training dynamics "handicap" easy spurious features and "boost" rare informative ones.

However, this non-linear feature learning has a dark side. In Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize, Tianren Zhang, Feng Chen, and others revealed that networks can learn background features completely uncorrelated with the label. These features "contaminate" the model, causing performance to crash when the environment changes. This highlights why shallow theory is often insufficient: in deep networks, these contaminated features can be passed up through layers, compounding errors in ways that two-layer models cannot capture.

The Depth Barrier

Despite the success of shallow network theory, a significant gap remains. Most rigorous proofs—whether for solving Schrödinger equations by Jianfeng Lu or achieving optimal adversarial error by Justin D. Li and Matus Telgarsky—are restricted to shallow architectures.

As Marc Barbier notes in Generalization performance of narrow one-hidden layer networks in the teacher-student setting, single-layer frameworks do not easily scale because depth introduces complex mathematical dependencies between units. While shallow networks can be highly expressive—for example, tanh networks can approximate smooth functions exponentially well—their inductive bias differs fundamentally from the hierarchical feature learning seen in deep ResNets or Transformers.

Bridging this gap requires moving beyond the "lazy training" or Neural Tangent Kernel (NTK) regime, where networks behave like linear models. Researchers like Taiji Suzuki are exploring how non-convex noisy gradient descent allows deep models to outperform kernel methods by adapting to the data's hidden structure, a property that shallow theory is only beginning to explain.

Papers in this version

Showing 60 of 364 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generalization_theory_for_shallow_and_de_256412d9,
  title = {Generalization Theory for Shallow and Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-theory-for-shallow-and-deep-neural-networks/rdr_256412d9b472439688cde779bf599a74},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalization_theory_for_shallow_and_de_256412d9_v1,
  title = {Generalization Theory for Shallow and Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-theory-for-shallow-and-deep-neural-networks/rdr_256412d9b472439688cde779bf599a74/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}