Abstract
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the collapsing domain problem, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks. We show this is a direct consequence of the Pythagorean equation. By noticing that any neural network is a residual network, we devise the residual trick, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.
Introduction
For years, the deep learning community has operated under the assumption that the "exploding gradient problem" was largely solved by a combination of Batch Normalization (BatchNorm), clever weight initialization (like He or Xavier), and adaptive optimizers like Adam. However, this paper argues that this is a dangerous misconception. The authors point out that the field has lacked a consistent, scale-invariant way to measure "explosion," leading to "confounded" metrics that can be "fixed" by simple weight rescaling without addressing the underlying optimization difficulty.
Measuring Pathologies: The Gradient Scale Coefficient (GSC)
To rigorously analyze gradient behavior, the authors introduce the Gradient Scale Coefficient (GSC). Unlike standard gradient norms, the GSC is invariant to simple weight rescaling. It measures the magnitude of the gradient flowing backward relative to the activations flowing forward. If the GSC grows exponentially with depth, the network suffers from exploding gradients, making it increasingly difficult to train as depth increases.
The Exploding vs. Collapsing Tradeoff
A central contribution of this work is identifying a fundamental tension in network design. If layers are unconstrained, gradients explode. However, if they are constrained too aggressively to prevent explosion, the network falls into the "collapsing domain problem."
In a collapsing domain, the activations for different inputs become so similar that the non-linearities (like ReLU) lose their representational power. The network becomes "pseudo-linear": a 50-layer network might effectively have the capacity of only a 2-layer network because the signal is trapped in the linear region of the activation functions. As shown in Figure 1, architectures like BatchNorm-ReLU exhibit exponential GSC growth (A), while others that suppress GSC growth often see a sharp rise in domain bias (D) and a drop in linear approximation error (F), signifying collapse.
Why ResNets Work: The Pythagorean Connection
The paper provides a geometric explanation for the success of Residual Networks (ResNets). ResNets circumvent explosion through a "dilution" effect. Because skip connections add an identity matrix to the residual Jacobian, the growth of the gradient is governed by the Pythagorean theorem rather than a simple product. This leads to much slower, sub-exponential growth compared to vanilla networks.
The Residual Trick and Effective Depth
By noticing that any neural network can be viewed as a residual network—where the initial state is an "initial function" and training updates are "residual functions"—the authors apply ResNet analysis to vanilla MLPs. This reveals the concept of Effective Depth: a measure of how many layers actually contribute to the computation. Most exploding architectures reach an effective depth of only 10–30, even with 50 physical layers.
To address these issues, the authors propose Looks-Linear Initialization (LLI) for ReLU networks. LLI ensures the initial state is approximately an orthogonal transformation, mimicking ResNet behavior. Experiments on CIFAR-10 show that a plain ReLU network using LLI can outperform a ResNet, suggesting that the initial mathematical state is often more critical than the architectural choice itself.
Conclusion
Exploding gradients are not just a numerical quirk but a structural limitation. While BatchNorm and SELU are popular, they do not inherently solve the explosion problem as measured by the GSC. The success of ResNets lies in their mathematical simplicity and their ability to balance the tradeoff between gradient explosion and domain collapse. Understanding these pathologies allows for better initialization strategies, like LLI, which can unlock the potential of deep architectures without relying solely on skip connections.
Summary updated