In classical optimization, adding complexity usually makes a problem harder to solve. In deep learning, however, we often observe the opposite: deeper and wider networks frequently converge faster and more reliably than their shallower, narrower counterparts. This phenomenon suggests that the "difficulty" of optimization is not an inherent property of depth, but rather a consequence of how signals propagate through the architecture.
Recent theoretical work has identified a specific regime where optimization becomes decoupled from depth. By combining over-parameterization (width) with residual connections, we can transform a daunting non-convex landscape into one that behaves as predictably as a simple linear model.
Width as a Convexifier
The most direct way to simplify the optimization landscape is to increase the network's width. As Simon S. Du and colleagues demonstrate in Width Provably Matters in Optimization for Deep Linear Neural Networks, sufficiently wide networks exhibit a linear convergence rate. This means the training loss decreases geometrically, effectively turning the training process into something as well-behaved as linear regression.
Intuition suggests that as width increases, the "Neural Tangent Kernel" (NTK)—a matrix describing how model outputs change with respect to parameters—becomes increasingly stable. In the "infinite width" limit explored by Andrea Montanari in A Mean Field View of the Landscape of Two-Layers Neural Networks, the training dynamics simplify into a predictable "mean field" or fluid flow. In this regime, adding more neurons doesn't complicate the landscape; it provides more "directions" for gradient descent to find a path to the global minimum, ensuring that performance becomes independent of the specific number of neurons once a threshold is met.
The Residual Shortcut to Polynomial Scaling
While width helps, standard feedforward networks face a "depth tax": to maintain trainability, the required width must grow exponentially with the number of layers. This is the "degradation problem" described in ResNet: Enabling Deep Convolutional Neural Networks through Residual Learning by Xingyu Liu and Kun Ming Goh.
Residual connections (skip connections) fundamentally change this scaling law. Simon Shaolei Du proves in Gradient Descent Finds Global Minima of Deep Neural Networks that ResNets only require width to grow polynomially with depth to guarantee convergence. By allowing gradients to bypass layers via identity mappings, ResNets ensure that the optimization landscape remains favorable even as the architecture grows extremely deep. This architectural choice effectively decouples the depth of the model from the difficulty of finding a global minimum.
Stabilizing the Signal Flow
Even with residuals and width, deep networks can suffer from signal explosion or vanishing gradients if not properly balanced. Researchers have identified three key mechanisms to stabilize this flow:
- Sharp Scaling Factors: Huishuai Zhang and others in Stabilize Deep ResNet with A Sharp Scaling Factor τ show that scaling the residual branch by exactly $\tau = 1/\sqrt{L}$ (where $L$ is the number of blocks) is a "sharp" threshold for stability. This scaling makes the optimal learning rate independent of depth, allowing 1000-layer networks to be trained with the same hyperparameters as 10-layer ones.
- Orthogonal Initialization: Standard Gaussian initialization requires width to increase linearly with depth to stay trainable. However, Lechao Xiao demonstrates in Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks that orthogonal initialization allows deep networks to converge efficiently regardless of width, providing a "depth-independent" design principle.
- The ReLU Free Lunch: Nonlinearities like ReLU actually help conditioning. Chaoyue Liu argues in Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks that ReLUs increase the separation between data points in the gradient space. This improves the condition number of the NTK, accelerating convergence as the network gets deeper.
The Convergence of Theory and Practice
The shared insight across these papers is that the "bottleneck" of depth is avoidable. When a network is sufficiently over-parameterized, the convergence rate is determined by the number of iterations rather than the width or depth of the model. As Zhiyan Ding shows in Overparameterization of deep ResNet: zero loss and mean-field analysis, in the limit where both width and depth go to infinity, gradient descent behaves like a predictable fluid flow toward a global minimum.
By carefully choosing initialization, scaling factors, and connectivity, we can reach a regime where depth is no longer an obstacle to optimization, but rather a tool for better representation and generalization. Readers should view depth not as a source of complexity, but as a feature that—when supported by adequate width—actually facilitates the path to a solution.