In classical optimization, non-convex functions are notoriously difficult to handle. They are typically riddled with spurious local minima—pockets in the loss surface where the gradient drops to zero, trapping optimization algorithms far away from the global optimum.
Yet, deep learning routinely defies this grim reality. Standard gradient descent reliably trains massive neural networks to near-zero training error. The resolution to this paradox lies in the geometry of width. As we add hidden units to a neural network, we are not just adding parameters; we are fundamentally altering the topology of the loss landscape. Over-parameterization acts as a landscape smoother, systematically dismantling optimization traps.
The Escape Hatch: Turning Minima into Saddles
To understand why width simplifies optimization, imagine being trapped at the bottom of a steep, two-dimensional valley. To escape, you would have to climb uphill. This is a local minimum. But what if a third dimension suddenly opened up, slicing through the valley wall and sloping downward? The local minimum would instantly transform into a saddle point, and you could continue your descent.
This is precisely what happens when a network's width is increased. Replicating or adding hidden units introduces redundant degrees of freedom. As shown by Fukumizu et al. in Semi-flat minima and saddle points by embedding neural networks to overparameterization, this embedding process creates new downward paths in the error landscape, destabilizing previously stable, suboptimal solutions.
As a network expands, these extra dimensions systematically convert troublesome, non-global minima into escapable saddle points. Yehudai et al. demonstrate in The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks that while adding neurons destroys local convexity around global minima, it simultaneously creates descent directions at previously stagnant points. Similarly, Arjevani et al. show in Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II that passing a certain width threshold transforms trapping spurious minima back into unstable saddle points, allowing gradient-based methods to bypass traps that would otherwise hinder smaller networks.
[Suboptimal Local Minimum] ──(Increase Width)──> [Unstable Saddle Point] ──> [Continued Descent]
The Threshold of Global Optimality
If adding a few neurons creates escape hatches, what happens when a network becomes sufficiently wide? The landscape undergoes a complete structural transformation: spurious local basins are entirely eliminated, leaving a "benign" landscape where every local minimum is a global minimum.
This phenomenon depends heavily on the relationship between the network's width and the complexity of the data. In Optimization Landscape and Expressivity of Deep CNNs, Nguyen et al. demonstrate that when a neural network layer's width exceeds the number of training samples, the loss surface is smoothed to the point where almost every critical point corresponds to a global minimum with zero training error. This structural guarantee holds true for deep convolutional architectures as well; as long as a sufficiently wide layer is followed by a fully connected layer, suboptimal basins vanish (The loss surface and expressivity of deep convolutional neural networks).
This elimination of "bad basins" is a fundamental property of wide architectures. Li et al. prove in On the Benefit of Width for Neural Networks: Disappearance of Basins that increasing the width of the last hidden layer relative to the number of data samples eliminates sub-optimal local minima regardless of the choice of activation function. Whether utilizing quadratic activations (Soltanolkotabi et al. in Theoretical insights into the optimization landscape of over-parameterized shallow neural networks; Sarao Mannelli et al. in Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions) or standard ReLUs (Safran et al. in Spurious Local Minima are Common in Two-Layer ReLU Neural Networks), over-parameterization guarantees that the empirical loss surface becomes globally well-behaved.
Annihilation via Mild Over-parameterization
A common misconception is that these geometric benefits require infinite width. In reality, the "annihilation" of spurious minima can occur with surprisingly little extra capacity.
In Annihilation of Spurious Minima in Two-Layer ReLU Networks, Arjevani et al. show that eliminating specific classes of spurious minima does not demand an infinitely wide model. Instead, adding a small, finite number of hidden units—sometimes just one or two—is mathematically sufficient to break symmetries and turn a trapping local minimum into an unstable saddle point. This concept of "mild" over-parameterization, also explored in tensor optimization contexts by Zhang et al. in Recovery Guarantees For Quadratic Tensors With Sparse Observations, highlights that the landscape-smoothing effect kicks in long before reaching asymptotic limits.
Smooth Paths and Global Convergence
The topological flattening of the loss surface fundamentally alters training dynamics. When saddle points are lowered and bad basins disappear, the optimization path becomes highly predictable, yielding several key benefits:
- Linear Convergence Rates: For deep linear networks, Du et al. prove in Width Provably Matters in Optimization for Deep Linear Neural Networks that sufficient width allows standard gradient descent with random initialization to converge to the global optimum at a linear rate, effectively turning a non-convex problem into one as well-behaved as simple linear regression.
- Topological Connectivity: Using topological data analysis, Barannikov et al. map loss surface "barcodes" in Barcodes as Summary of Loss Function Topology, confirming that increasing width lowers the relative height of saddle points. This creates a highly connected landscape where disparate low-loss solutions are linked by continuous paths of low error, explaining why stochastic gradient descent reliably finds high-quality solutions (Shevchenko et al. in Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks).
- Robustness Across Domains: This width-induced smoothing is not unique to standard supervised learning. In deep reinforcement learning, over-parameterized policy networks operating in the mean-field regime bypass non-convexity entirely, allowing policy gradient algorithms to reliably discover globally optimal policies (Agazzi et al. in Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime; Liu et al. in Neural Trust Region/Proximal Policy Optimization Attains Globally Optimal Policy). The same mechanism eliminates spurious traps in expectation-maximization for mixture models (Xu et al. in Benefits of over-parameterization with EM) and guarantees generalization on linearly separable data (Brutzkus et al. in SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data).
Ultimately, over-parameterization resolves the tension between expressivity and optimization. By making a network wider, we do not make the optimization problem more complex; instead, we leverage high-dimensional geometry to smooth away the rugged traps of the non-convex landscape.
Key Foundational Work
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks (2017) — contributed 1 concept elements to this direction.
Computing with Arrays of Bell-Shaped and Sigmoid Functions (1990) — contributed 1 concept elements to this direction.