Abstract
We study the supervised learning problem under either of the following two models: (1) Feature vectors $x_i$ are $d$-dimensional Gaussian and responses are $y_i = f_*(x_i)$ for $f_*$ an unknown quadratic function; (2) Feature vectors $x_i$ are distributed as a mixture of two $d$-dimensional centered Gaussians, and $y_i$'s are the corresponding class labels. We use two-layers neural networks with quadratic activations, and compare three different learning regimes: the random features (RF) regime in which we only train the second-layer weights; the neural tangent (NT) regime in which we train a linearization of the neural network around its initialization; the fully trained neural network (NN) regime in which we train all the weights in the network. We prove that, even for the simple quadratic model of point (1), there is a potentially unbounded gap between the prediction risk achieved in these three training regimes, when the number of neurons is smaller than the ambient dimension. When the number of neurons is larger than the number of dimensions, the problem is significantly easier and both NT and NN learning achieve zero risk.
Introduction: The Lazy Training Paradigm
Recent theoretical breakthroughs in deep learning, such as the Neural Tangent Kernel (NTK), suggest that very wide neural networks behave like simple linear models during training. In this "lazy" regime, the network stays close to its initialization, effectively acting as a linear model over fixed features. While this perspective simplifies mathematical analysis, it ignores the core intuition that deep networks succeed by learning better representations of data. This paper, presented at NeurIPS 2019, quantifies exactly what is lost when we treat a network as a lazy linear model.
Methods and Theoretical Framework
The researchers compare three training regimes for two-layer neural networks with quadratic activations ($\sigma(u) = u^2$):
- Random Features (RF): Only the second-layer weights are trained; the first layer remains fixed at random.
- Neural Tangent (NT): The network is linearized around its initialization, and this linearization is trained (the "lazy" regime).
- Fully Trained (NN): All weights in the network are trained using Stochastic Gradient Descent (SGD).
To make the comparison mathematically precise, the authors use high-dimensional asymptotics, letting the number of neurons ($N$) and input dimensions ($d$) both go to infinity while keeping their ratio ($\rho = N/d$) constant.
The $N < d$ Bottleneck and Feature Learning
The most critical finding is the performance bottleneck that occurs when the number of neurons $N$ is smaller than the input dimension $d$. Lazy models (RF and NT) are trapped by their initialization; they can only "see" the data through a random $N$-dimensional slice of the $d$-dimensional space.
In contrast, a fully trained NN can "rotate" its neurons to align with the most important directions of the target function—specifically the principal eigenvectors of the target quadratic form. This adaptive feature learning allows the NN to capture the relevant signal even with a limited number of neurons, leading to a potentially unbounded gap in prediction risk compared to NT models.
Key Results and Empirical Validation
The paper provides sharp, closed-form expressions for the prediction risk in all three regimes. For the quadratic model with $N < d$, the NN risk depends on the "tail" of the target matrix's eigenvalues, whereas the NT risk depends on a random average of all eigenvalues.
Numerical simulations perfectly match these theoretical predictions. As shown in the figures, there is a clear "V-shape" or gap in the error plots where NN significantly outperforms NT and RF when $N < d$. While both NT and NN can achieve zero risk when $N > d$ (as random directions eventually span the entire space), NN reaches lower risk levels much faster as $N$ increases.
Conclusion
This research demonstrates that lazy training regimes fail to capture the representational power of neural networks in the over-parameterized but neuron-limited regime ($N < d$). While the NTK perspective is theoretically convenient, it does not fully explain the success of deep learning in scenarios where feature learning is essential. Future work will explore whether these performance gaps persist in deeper architectures and with more complex activation functions like ReLU.
Summary updated