In classical optimization theory, non-convexity is a death sentence. A non-convex landscape is typically imagined as a "minefield" of local minima and saddle points where gradient descent should, theoretically, get trapped. Yet, in deep learning, simple first-order methods like Stochastic Gradient Descent (SGD) consistently find global optima.
This discrepancy—the "Deep Learning Mystery"—has led to a massive research effort to characterize the geometry of neural network loss surfaces and prove why they are far more tractable than worst-case complexity analysis suggests.
Overparameterization as a Convexifier
The most prominent explanation for why these models converge is overparameterization. When a network has many more parameters than data points, the optimization landscape undergoes a qualitative shift.
Simon Shaolei Du and colleagues demonstrated in Gradient Descent Finds Global Minima of Deep Neural Networks that if a network is sufficiently wide, the loss landscape becomes "nice" enough for gradient descent to reach zero training error. This is often explained through the Neural Tangent Kernel (NTK) or Mean-Field regimes, where, as Cong Fang and Tong Zhang argue in Mathematical Models of Overparameterized Neural Networks, overparameterization effectively "convexifies" the landscape.
Crucially, this doesn't require infinite width. Mahdi Soltanolkotabi and Samet Oymak showed in Towards moderate overparameterization: global convergence guarantees for training shallow neural networks that global convergence is guaranteed even with "moderate" overparameterization—roughly when the number of parameters scales with the square of the number of data points.
The Absence of Spurious Valleys
If the landscape were truly riddled with "bad" local minima (spurious valleys), optimization would stall. However, research into the intrinsic dimension of these networks suggests these traps are rare or non-existent in high-dimensional spaces.
Afonso S. Bandeira and Joan Bruna proved in Neural Networks with Finite Intrinsic Dimension have no Spurious Valleys that once a network exceeds a certain width, these valleys vanish. Instead of a minefield, the landscape looks more like a collection of "basins" leading to flat, degenerate global minima. Tomaso Poggio and Qianli Liao argue in Theory II: Landscape of the Empirical Risk in Deep Learning that these flat regions are not just easy to find, but are actually the dominant feature of the landscape.
Symmetry Breaking and Dynamical Systems
Even if the landscape is "nice," the starting point matters. Many neural networks possess inherent symmetries (e.g., you can swap two neurons in a hidden layer without changing the output). These symmetries create "saddle points" where the gradient is zero, but the solution is sub-optimal.
Yuandong Tian explored this in Symmetry-Breaking Convergence Analysis of Certain Two-layered Neural Networks with ReLU nonlinearity. By treating training as a dynamical system, Tian proved that while symmetric initializations lead to dead ends, "symmetry-breaking" initializations allow the network to escape these saddles and converge to the global optimum. This explains why random initialization is so effective: it naturally breaks these symmetries.
Architectural Facilitators: ResNets and Transformers
The architecture itself often dictates how easy the landscape is to navigate. A recurring theme is that certain structures "smooth" the optimization path.
- Skip-connections: Weinan E and Chao Ma showed in Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections that ResNets achieve global minima exponentially faster than standard feedforward networks because the skip-connections prevent the gradients from vanishing or becoming too chaotic.
- Attention Mechanisms: In the context of large-scale models, Cheng Gao and Yuan Cao proved in Global Convergence in Training Large-Scale Transformers that Transformers possess a specific geometry that allows gradient descent to bypass local minima traps that would otherwise hinder convergence.
Implicit Regularization and the Path to Generalization
Finally, the research suggests that we don't just find any global minimum, but good ones. The optimization process itself has an implicit bias.
Nadav Cohen and Noam Razin demonstrate in Lecture Notes on Linear Neural Networks that depth acts as an implicit preconditioner, accelerating the search for low-rank, simple solutions. Similarly, Masaaki Imaizumi and Johannes Schmidt-Hieber argue in On Generalization Bounds for Deep Networks Based on Loss Surface Implicit Regularization that the local geometry of the loss landscape traps SGD in "stagnation sets" that prevent overfitting, explaining why these massive models generalize so well despite their complexity.