On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions

Martín Simón, Francis Bach, Giulio Biroli

Abstract

We study the training dynamics of a shallow neural network with quadratic activation functions and quadratic cost in a teacher-student setup. In line with previous works on the same neural architecture, the optimization is performed following the gradient flow on the population risk, where the average over data points is replaced by the expectation over their distribution, assumed to be Gaussian. We first derive convergence properties for the gradient flow and quantify the overparameterization that is necessary to achieve a strong signal recovery. Then, assuming that the teachers and the students at initialization form independent orthonormal families, we derive a high-dimensional limit for the flow and show that the minimal overparameterization is sufficient for strong recovery. We verify by numerical experiments that these results hold for more general initializations.

Introduction

Despite the massive success of deep learning, explaining why gradient-based optimization succeeds on highly non-convex loss landscapes remains a central challenge. Theoretically, such landscapes are expected to contain numerous "spurious local minima" that could trap training before a global optimum is reached. This paper investigates a fundamental question: how much overparameterization is required to eliminate these traps?

The authors employ a "teacher-student" framework to analyze shallow networks (one hidden layer) with quadratic activation functions in high-dimensional settings. By focusing on the regime where the number of features $d$ is large, they provide a precise mathematical threshold for successful training.

Methodology

The study focuses on the gradient flow on population risk, analyzing the idealized case of infinite data where inputs follow a Gaussian distribution $\mathcal{N}(0, I_d)$.

Key Technical Components

  • Quadratic Activations: The use of $x^2$ activations allows for tractable analytical solutions while preserving the non-convex nature of the weights.
  • High-Dimensional Limit: The analysis considers the regime where the dimension $d$, the number of student neurons $m$, and the number of teacher neurons $m^*$ all go to infinity, while their ratios $\alpha = m/d$ and $\alpha^* = m^*/d$ remain fixed.
  • Random Matrix Theory: The authors utilize properties of the $\beta$-Jacobi ensemble to understand the spectral distribution of the weight matrices during training.
Spectral distribution of the weights at $d=2000$ compared to the theoretical $d=\infty$ limit.

The figure above demonstrates how the empirical spectral distribution of the weights for a finite dimension ($d=2000$) closely matches the theoretical prediction for the infinite-dimensional limit, justifying the use of high-dimensional analysis.

Theoretical Insights

  • The $m \ge m^*$ Threshold: A key finding is that a student network does not require massive overparameterization. As long as the student has at least as many neurons as the teacher ($m \ge m^*$), gradient flow almost surely converges to the global minimum, achieving perfect signal recovery.
  • Convexity in Disguise: Although the loss function is non-convex with respect to the weight matrix $W$, it is convex with respect to the product $WW^T$. This underlying structure is critical for bypassing local minima.
  • Convergence Rates: While more neurons ($m > m^*$) guarantee recovery, they can slow down convergence. The loss decays as $O(1/t^2)$ when $m > m^*$, compared to potentially exponential decay when $m = m^*$.

Results and Discussion

The authors prove that under mild assumptions, gradient flow always converges to a global minimizer. In high dimensions, a "recovery phase transition" occurs: the overlap between student and teacher reaches 1 if $\alpha \ge \alpha^*$, whereas only partial recovery is possible if $\alpha < \alpha^*$.

Evolution of the potential function $\phi_d(\gamma)$ across different overparameterization regimes.

As shown in the figure above, the deterministic equations derived by the authors (dashed lines) perfectly predict the training trajectories observed in numerical simulations (colored lines). The three panels illustrate the distinct behaviors in the overparameterized ($\alpha > \alpha^*$), critically parameterized ($\alpha = \alpha^*$), and underparameterized ($\alpha < \alpha^*$) regimes.

Conclusion

This work provides a rigorous foundation for understanding overparameterization in shallow networks. It demonstrates that the transition from training failure to guaranteed success occurs precisely when the student network's complexity matches that of the teacher. Future work will focus on extending these results to more general activation functions like ReLU and analyzing the impact of finite datasets on the empirical loss landscape.

Summary updated

Abstract

We study the training dynamics of a shallow neural network with quadratic activation functions and quadratic cost in a teacher-student setup. In line with previous works on the same neural architecture, the optimization is performed following the gradient flow on the population risk, where the average over data points is replaced by the expectation over their distribution, assumed to be Gaussian.We first derive convergence properties for the gradient flow and quantify the overparameterization that is necessary to achieve a strong signal recovery. Then, assuming that the teachers and the students at initialization form independent orthonormal families, we derive a high-dimensional limit for the flow and show that the minimal overparameterization is sufficient for strong recovery. We verify by numerical experiments that these results hold for more general initializations.
BibTeX
@inproceedings{simon2024impact,
  author = {Martín Simón and Francis Bach and Giulio Biroli},
  title = {{On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=4qAgMiQSBE},
}
Figure