Deep learning theory faces a persistent dilemma. While neural networks excel at processing highly structured, real-world data, proving why they converge or generalize often requires assuming the data is completely unstructured—specifically, that inputs follow an i.i.d. standard Gaussian distribution.
This "Gaussian input assumption" is the bedrock of classical proofs for optimization landscapes, convergence rates, and generalization bounds. Yet, it creates a significant gap between mathematical tractability and practical reality.
Why the Gaussian Assumption is a Mathematical Superpower
To understand how to move beyond Gaussian inputs, we must first understand why theorists rely on them so heavily. The assumption provides three primary mathematical advantages:
1. Rotational Invariance and Closed-Form Gradients
When inputs are spherically symmetric, the expected gradient over the data distribution (the population gradient) can often be computed analytically. For instance, Yuandong Tian in An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis derived a rare closed-form formula for the population gradient of a two-layer ReLU network by exploiting spherical Gaussian inputs, revealing flat manifolds of critical points. Without this symmetry, tracking the trajectory of gradient descent becomes an intractable dynamical system.
2. The Magic of Orthogonal Polynomials
Under a standard Gaussian distribution, activation functions like ReLUs can be cleanly decomposed using Hermite polynomials. As shown by Sitan Chen et al. in Learning Narrow One-Hidden-Layer ReLU Networks, these polynomial expansions, combined with Gaussian anti-concentration properties, allow algorithms to provably learn narrow one-hidden-layer networks. If the distribution shifts to something as simple as a uniform or discrete distribution, these algebraic properties disintegrate.
3. Simplifying Optimization Landscapes
The assumption that inputs are well-behaved Gaussians eliminates many pathological geometric features of the loss landscape. Mahdi Soltanolkotabi demonstrated in Learning ReLUs via Gradient Descent that gradient descent can successfully learn a single ReLU unit when inputs are i.i.d. Gaussian, bypassing the non-convex traps that theoretically plague local search heuristics. Similarly, Kai Zhong et al. in Recovery Guarantees for One-hidden-layer Neural Networks leveraged Gaussianity to provide global recovery guarantees for one-hidden-layer networks given a proper initialization.
Where the Assumption Breaks Down
While Gaussian inputs make the proofs elegant, they fail to capture the statistical realities of natural datasets.
1. Structural and Spatial Dependencies
Real-world data, such as natural images, possesses intense spatial correlations. When analyzing convolutional neural networks (CNNs), assuming i.i.d. Gaussian inputs ignores the very spatial structures that CNNs are designed to exploit. Jason D. Lee et al. in Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima highlighted this gap, proving that while gradient descent can navigate spurious local minima in shallow CNNs under Gaussian inputs, these guarantees do not naturally extend to complex spatial dependencies.
2. The Reality of Classification
Many theoretical frameworks treat deep learning as a teacher-student regression problem over centered data. However, HEE BIN YOO et al. in How Classifier Features Transfer to Downstream: An Asymptotic Analysis in a Two-Layer Model point out that realistic classification tasks involve non-centered data where distinct classes possess structured, multi-modal distributions. A theory built entirely on centered Gaussians cannot accurately predict how features transfer to downstream tasks in these settings.
3. Deceptive Loss Landscapes
Spurious local minima—local optima that are not globally optimal—are highly sensitive to the input distribution. Yossi Arjevani et al. in Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II noted that analytical characterizations of these minima heavily rely on Gaussianity, leaving open the possibility that non-Gaussian data alters the optimization landscape in ways current theory cannot foresee.
Pathways to General Input Distributions
To bridge the gap between theory and practice, recent work focuses on relaxing or entirely replacing the Gaussian input assumption.
Broadening the Distribution Class
Instead of demanding strict Gaussianity, researchers are extending proofs to broader families like log-concave distributions. For example, Pranjal Awasthi et al. in Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations developed sample-efficient algorithms for depth-2 ReLU networks with bias terms under log-concave distributions. To handle even more general settings, Daniel Kane et al. in Efficiently Learning One-Hidden-Layer ReLU Networks via SchurPolynomials introduced an algorithm using Schur polynomials, pushing the boundaries of what can be learned efficiently without standard Gaussian constraints.
Score Function Estimation
Rather than modifying the network architecture, one can modify the loss function to accommodate arbitrary data. Ashok Vardhan Makkuva et al. in Learning One-hidden-layer Neural Networks under General Input Distributions introduced the Local Likelihood Score Function Estimator (LLSFE). By embedding the input distribution's score function directly into the loss, the authors designed an optimization landscape completely free of bad local minima for a wide range of non-Gaussian, multi-modal, or heavy-tailed distributions.
From Probabilistic to Geometric Guarantees
Perhaps the most radical shift is abandoning probabilistic distribution models altogether in favor of pure geometry. Instead of assuming data comes from a specific distribution, Kedar Karhadkar et al. in Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension provided tight bounds on the smallest eigenvalue of the Neural Tangent Kernel (NTK) based purely on the geometric separation of data points on a sphere. This ensures trainability and memorization for arbitrary data, regardless of its underlying statistical distribution.
Mild "Spread-Out" Conditions
Even when analyzing a single neuron, moving away from Gaussianity requires new tools. Gilad Yehudai et al. in Learning a Single Neuron with Gradient Methods proved general convergence guarantees for gradient methods under mild, "spread-out" distribution conditions, demonstrating that strict Gaussianity is a convenience rather than a fundamental requirement for learning simple non-linear units.
Key Foundational Work
History-Dependent Attractor Neural Networks (1992) — contributed 1 concept elements to this direction.
Learning with Noise and Regularizers in Multilayer Neural Networks (1996) — contributed 1 concept elements to this direction.