Abstract
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the collapsing domain problem, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks. We show this is a direct consequence of the Pythagorean equation. By noticing that any neural network is a residual network, we devise the residual trick, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.
Introduction
Training deep neural networks is fundamentally limited by the stability of signal propagation. While the deep learning community often assumes that Batch Normalization, careful initialization (He/Glorot), and adaptive optimizers like Adam have "solved" the vanishing and exploding gradient problems, this paper demonstrates that these pathologies persist in modern architectures. By introducing a new metric, the Gradient Scale Coefficient (GSC), we show that gradients often explode exponentially with depth, severely limiting the "effective depth" of networks—the number of layers that actually contribute to learning.
Methodology: Measuring Gradient Pathology
To rigorously analyze these issues, the authors introduce two key conceptual tools:
- Gradient Scale Coefficient (GSC): Traditional gradient norms are sensitive to simple weight rescaling. The GSC measures the backward gradient magnitude relative to forward activations, providing a scale-invariant metric of gradient stability.
- The Residual Trick: By mathematically reformulating any standard feedforward network as a residual network, the authors can measure "effective depth." This allows for a direct comparison between vanilla architectures and ResNets using the same analytical framework.
Theoretical analysis was performed using Gaussian noise to observe signal propagation, while empirical validation was conducted on the CIFAR-10 dataset to track optimization dynamics.
The Myth of Solved Gradients
Empirical analysis reveals that many popular architectures—including those using Batch Normalization, Layer Normalization, or SELU—still suffer from exponential gradient explosion at initialization.
As shown in Figure 1(A), the GSC grows exponentially with compositional depth for several common configurations. For instance, a 50-layer MLP with Batch Normalization and ReLU can exhibit a GSC in the thousands. This explosion forces optimizers to use extremely small learning rates, effectively "shallowing" the network as deep layers fail to co-adapt.
The Collapsing Domain Problem
Architectures designed to avoid exploding gradients often fall into a different trap: the collapsing domain. When gradients are stabilized, activations may cluster so tightly that non-linearities like ReLU or tanh operate in a near-linear regime. Figure 1(D) and 1(E) illustrate this through "pre-activation domain bias" and "sign diversity." In these cases, the network loses its expressive power, effectively becoming a deep linear transformation that cannot learn complex functions.
Why ResNets Succeed: The Dilution Effect
The success of ResNets is often attributed to "gradient flow," but this paper offers a more precise mathematical explanation: dilution. A skip connection adds a linear identity mapping to the non-linear residual block. Because these two paths are largely uncorrelated at initialization, the Pythagorean theorem implies that the variance of the sum is the sum of the variances. This mathematically "dilutes" the growth of the GSC, keeping gradients stable even at extreme depths.
Results and the Orthogonal Initial State (OIS)
The authors' experiments confirm these theoretical insights:
- Effective Depth Limits: Vanilla 50-layer networks often reach an effective depth of only 10-20 layers due to gradient instability.
- ResNet Superiority: ResNet counterparts maintain stable GSCs and achieve significantly lower training error. For example, a vanilla Batch-ReLU network had an initialized GSC of ~5,728, while its ResNet counterpart maintained stability.
- The OIS Alternative: By using a "looks-linear" initialization to achieve an Orthogonal Initial State (OIS), a vanilla ReLU network can achieve a perfect GSC of 1.0. Remarkably, such a network can outperform ResNets on training error (0.002% vs 0.06%), proving that the architectural benefit of ResNets can be replicated in vanilla networks through superior initialization.
Conclusion
Exploding gradients remain a primary bottleneck in deep learning, even with modern normalization techniques. ResNets succeed not by some mystical property, but by mathematically simplifying the network's initial state. Understanding the balance between gradient explosion and domain collapse is essential for designing the next generation of truly deep architectures.
Summary updated