Abstract
We study the training dynamics of a shallow neural network with quadratic activation functions and quadratic cost in a teacher-student setup. In line with previous works on the same neural architecture, the optimization is performed following the gradient flow on the population risk, where the average over data points is replaced by the expectation over their distribution, assumed to be Gaussian. We first derive convergence properties for the gradient flow and quantify the overparameterization that is necessary to achieve a strong signal recovery. Then, assuming that the teachers and the students at initialization form independent orthonormal families, we derive a high-dimensional limit for the flow and show that the minimal overparameterization is sufficient for strong recovery. We verify by numerical experiments that these results hold for more general initializations.
Introduction
A fundamental mystery in deep learning is why non-convex optimization problems, which should theoretically be riddled with trapping local minima, are successfully solved by simple gradient-based methods. The prevailing hypothesis is that overparameterization—using more neurons than the data's intrinsic dimension—smooths the loss landscape. However, many theoretical guarantees require an impractical number of neurons. This paper investigates the minimal overparameterization necessary for global convergence in high-dimensional settings.
Methodology
The authors employ a teacher-student setup, where a student network with $m$ neurons attempts to learn the weights of a teacher network with $m^*$ neurons.
- Architecture: The study focuses on one-hidden-layer networks with quadratic activation functions ($x^2$). This choice preserves the non-convex nature of the problem while remaining mathematically tractable.
- Optimization: The analysis centers on Gradient Flow on the population risk, assuming inputs are drawn from a Gaussian distribution $\mathcal{N}(0, I_d)$.
- High-Dimensional Limit: Using tools from statistical physics, the authors derive deterministic equations that describe the system as the dimension $d$ and the number of neurons $m, m^*$ grow to infinity at fixed ratios ($\alpha = m/d$).
Key Insights
The Minimal Threshold for Recovery
The research demonstrates that global convergence does not require $m$ to be proportional to $d$. Instead, the "minimal" threshold is simply $m \ge m^*$. Once the student has at least as many neurons as the teacher, spurious local minima effectively vanish or become avoidable.
The Overlap Metric
The alignment between student and teacher weights is quantified by the overlap metric ($\chi$). The study identifies a sharp phase transition: if $m < m^*$, only partial signal recovery is possible ($\sqrt{m/m^*}$); if $m \ge m^*$, perfect recovery ($\chi = 1.0$) is achieved.
The Convergence Paradox
Interestingly, while more neurons facilitate finding the global minimum, they can impede the speed of convergence.
- When $m = m^*$, convergence is exponentially fast.
- When $m > m^*$ (overparameterized), the rate slows to a power law ($1/t^2$). This slowdown is attributed to "flat directions" in the loss landscape created by the redundant neurons.
Key Results and Empirical Validation
Numerical experiments show that the high-dimensional theory matches finite-dimensional simulations remarkably well, even for relatively small $d$.
The results confirm that gradient flow almost surely converges to a global minimizer if $m \ge \min(m^*, d)$. The derived self-consistent equations accurately predict the entire training trajectory.
Limitations and Future Work
While providing rigorous insights, the study has limitations:
- Quadratic Activations: These lack the saturation or "dead zones" characteristic of Sigmoid or ReLU functions.
- Population Risk: The assumption of infinite data ignores the noise introduced by finite empirical samples.
- Gaussian Data: The proofs rely on rotational invariance. While numerical evidence suggests broader applicability, formal proofs for non-Gaussian distributions are still needed.
Summary updated