In the early days of deep learning theory, a comforting intuition emerged: if you make a neural network wide enough, the optimization landscape simplifies, and every local minimum becomes a global minimum. This "local-is-global" property is a hallmark of linear networks, which are surprisingly easy to optimize despite being non-convex. However, recent theoretical work has revealed that this property is incredibly fragile. The moment we introduce even the slightest nonlinearity—the very thing that makes deep learning powerful—the landscape shatters into a complex terrain of spurious local minima.
The Linear Ideal
To understand why nonlinear networks are difficult, we must first look at why linear networks are easy. In Critical Points of Linear Neural Networks: Analytical Forms and Landscape Properties, Yi Zhou and Yingbin Liang map the loss landscape of linear architectures. They demonstrate that while these landscapes are technically non-convex, they lack the "traps" found in deeper models.
Matthew Trager and Kathlén Kohn further clarify this in Pure and Spurious Critical Points: a Geometric Study of Linear Networks. They distinguish between "pure" critical points (intrinsic to the function) and "spurious" ones (artifacts of parameterization). In linear models, the absence of bad local minima is often a gift from the quadratic loss function itself; change the loss, and even linear networks can become tricky.
The Nonlinear Rupture
The transition from linear to nonlinear is not a smooth degradation; it is a fundamental phase shift. Chulhee Yun shows in Small nonlinearities in activation functions create bad local minima in neural networks that even a tiny deviation from linearity—a "small" nonlinearity—is enough to create spurious local minima.
This isn't just a quirk of specific functions like ReLU or Sigmoid. Ruoyu Sun and colleagues prove in Sub-Optimal Local Minima Exist for Almost All Over-parameterized Neural Networks that these traps exist for almost all nonlinear activations, regardless of how wide the network is. Over-parameterization, once thought to be a panacea for local minima, does not actually eliminate them.
Why Nonlinearity Creates Traps
The mechanism behind these traps is geometric. When we use piecewise linear activations like ReLU, we effectively partition the loss surface into a collection of "cells." Fengxiang He and Dacheng Tao explain in Piecewise linear activations substantially shape the loss surfaces of neural networks that these cells create an infinite number of spurious local minima.
Furthermore, these nonlinearities introduce "asymptotic" traps. Tengyu Xu and Yi Zhou note in When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models? that as weight norms increase, gradients can vanish in sub-optimal directions, effectively "freezing" the optimizer in a bad state.
Practical Implications
If nonlinearities are so problematic, why do we use them? The answer lies in representational power, but the cost is optimization complexity. Anders Øland and Bhiksha Raj argue in Be Careful What You Backpropagate that we should be more strategic about where we place these nonlinearities. They show that replacing the ubiquitous softmax with linear activations and manually boosting gradients can significantly speed up convergence by removing unnecessary curvature from the final layer.
Ultimately, the research suggests that the "messiness" of deep learning—the exploding weights, the sensitivity to initialization, and the struggle to converge—is a direct mathematical consequence of the non-convex, non-closed function spaces created by nonlinear neurons Philipp Petersen et al.. Understanding that these traps are an inherent property of nonlinearity, rather than a bug, is the first step toward designing more robust optimization strategies.