Deep learning presents a fundamental paradox: we optimize highly non-convex functions using simple gradient-based methods, yet we consistently find global minima that generalize well. For years, this was dismissed as an empirical mystery. However, a growing body of theoretical research is revealing that the "secret sauce" is overparameterization. By using far more parameters than data points, the optimization landscape of a neural network simplifies, allowing gradient descent to bypass what would otherwise be treacherous local minima.
The Overparameterization Paradox
In classical statistics, adding too many parameters leads to overfitting. In deep learning, however, increasing the width of a network often makes it easier to train and more robust. Mahdi Soltanolkotabi and Samet Oymak demonstrated in Towards moderate overparameterization that shallow networks don't need infinite width to converge. They proved that as soon as the number of parameters reaches a "moderate" threshold—roughly the square of the number of data points—gradient descent is guaranteed to find a global optimum.
Why does this happen? The intuition lies in the geometry of the loss landscape. Yingyu Liang and colleagues argued in Diversity Leads to Generalization in Neural Networks that if the hidden units of a network are sufficiently diverse, the training loss becomes proportional to the gradient. In such a landscape, there are no "spurious" local minima; every stationary point is a global one. Overparameterization essentially provides the "room" for this diversity to exist.
The Linear Proxy and Early Dynamics
One of the most powerful tools for understanding this convergence is the Neural Tangent Kernel (NTK). In the limit of infinite width, a neural network behaves like a linear model. Sanjeev Arora, Simon S. Du, and Wei Hu used this framework in Fine-Grained Analysis of Optimization and Generalization to show that training dynamics in overparameterized networks are surprisingly predictable.
This simplicity is most visible at the start of training. Lechao Xiao and Jeffrey Pennington proved in The Surprising Simplicity of the Early-Time Learning Dynamics that networks initially bias themselves toward the simplest possible linear solutions. This "spectral bias" ensures that low-frequency, generalizable patterns are mastered before the network begins to exploit its full non-linear capacity to fit complex noise.
The Frontier of Depth
While the theory for shallow (two-layer) networks is robust, deep networks introduce complex interactions between layers that are harder to track. Recent work has begun to bridge this gap. Simon Shaolei Du and Ming Ronnier Luo provided a breakthrough in Global Convergence of Four-Layer Matrix Factorization, proving that gradient descent can avoid saddle points even in deeper linear structures.
The real advantage of depth, however, is not just convergence but hierarchical learning. Yatin Dandi and Florent Krzakala showed in The Computational Advantage of Depth that deep networks act as progressive filters. They "coarse-grain" high-dimensional data into simpler representations across layers, a process that allows them to learn complex targets with far fewer samples than any shallow model could.
Feature Learning vs. Lazy Training
A critical distinction in this field is between "lazy training" (where weights barely move from their initialization) and "feature learning" (where the network develops new internal representations). Zhengdao Chen, Eric Vanden-Eijnden, and Joan Bruna have pushed beyond the NTK regime to provide global convergence guarantees for feature learning. They showed that by using "mean-field" scaling, networks can reach zero training loss while simultaneously building features that outperform standard kernel methods.
Beyond Supervised Learning
These convergence guarantees are now extending into Reinforcement Learning (RL) and Control. For instance, Zhaoran Wang and Hoi To Wai introduced Neural GTD, the first gradient temporal difference algorithm with neural networks that is provably guaranteed to find a global minimum in off-policy learning. Similarly, Stephen Tu and Jean-Jacques Slotine have leveraged these insights to create adaptive controllers that break the curse of dimensionality in non-linear systems.
The unifying theme across these 366 papers is clear: overparameterization is not a bug, but a feature. It transforms the daunting, non-convex optimization of deep networks into a manageable process that behaves with the reliability of linear models while retaining the expressive power of deep hierarchies.