Gradients explode - Deep Networks are shallow - ResNet explained

George Philipp, Dawn Song, Jaime G. Carbonell

For years, the deep learning community has operated under the assumption that techniques like Batch Normalization, specialized initializations (He/Glorot), and SeLU activations have largely "fixed" the exploding gradient problem. However, this paper challenges that notion, demonstrating that exploding gradients still plague many modern architectures and fundamentally limit a network's "effective depth."

The authors identify a major gap in how we evaluate neural networks: we often confuse nominal depth (how many layers we build) with effective depth (how many layers actually learn to work together). If gradients explode, a network becomes "effectively shallow," meaning it behaves like a much smaller ensemble of layers rather than a truly deep, integrated system.

Measuring the Problem: The Gradient Scale Coefficient

To rigorously analyze this, the researchers introduce several novel conceptual and mathematical tools:

  • Gradient Scale Coefficient (GSC): A new metric for measuring gradient explosion that is robust to simple "tricks" like rescaling weights. It measures the relative sensitivity of an early layer to changes in a later layer, normalized by the size of the activations.
  • Effective Depth: A formal measure of how many layers in a network are actually co-adapting during training. The authors provide theoretical proofs linking exploding gradients to an exponential increase in the number of updates required to reach a certain effective depth.
  • The Residual Trick: A clever mathematical framing that allows any neural network to be viewed as a residual network, enabling the same analysis to be applied to vanilla MLPs and ResNets alike.

The Twin Hurdles: Exploding Gradients and Domain Collapse

The most striking realization from the paper is that exploding gradients don't just make training unstable; they mathematically force the network to stay shallow. To reach a high "effective depth," a network with exploding gradients would require an exponentially large number of training steps.

When designing networks, engineers often face a frustrating trade-off:

  1. The Surjectivity Trade-off: Layers that are "surjective" (mapping inputs to the whole output space, like tanh or SeLU) tend to cause exploding gradients.
  2. The Collapsing Domain Problem: If you try to fix exploding gradients by making layers non-surjective (like ReLU), you often run into a different wall: the activations become too similar across different data points. This "domain collapse" also limits effective depth by making the network "pseudo-linear."
Gradient Scale Coefficient and Activation Diversity across layers

As seen in static analysis using Gaussian noise, popular architectures like SeLU and Batch-ReLU still exhibit massive gradient explosion under the GSC metric (Panel A), while non-surjective activations often suffer from collapsing activation diversity (Panels B and C).

Why ResNets Actually Work

If standard deep networks are effectively shallow, why do ResNets perform so well? The authors demonstrate that ResNets succeed not just by adding skip connections, but by mathematically simplifying the network to allow for much deeper co-adaptation.

ResNets work because their skip connections "dilute" the complex nonlinearities of each block with a simple identity mapping. This doesn't just reduce gradient growth linearly; it reduces it quadratically. This dilution allows the network to maintain a high effective depth even when the nominal depth is huge. For example, in experiments on CIFAR-10, a 51-layer vanilla MLP's effective depth often stalls around 10–15 layers, whereas ResNets reach an effective depth much closer to their nominal depth.

Interestingly, the architecture isn't the only way to solve this. The authors show that "Looks-Linear Initialization" (LLI), which starts a ReLU network in an approximately orthogonal state, can achieve lower training errors on CIFAR-10 than standard ResNets. This suggests that the initial state of the network is just as important as the architecture itself.

Limitations and Future Work

While the paper provides a rigorous foundation for understanding depth, it acknowledges a few boundaries:

  • Architecture Scope: The analysis primarily focuses on Multi-Layer Perceptrons (MLPs). While the principles likely apply to Convolutional (CNNs) and Recurrent (RNNs) networks, those architectures involve more complex "bookkeeping" not fully explored here.
  • Representational Capacity: "Diluting" a network (as ResNets do) might limit its ability to represent highly complex functions, but there is currently no mechanism to measure exactly how much dilution is "too much."
  • Practical Bounds: The theoretical bound for training time provided in the paper is noted as being "too loose" for practical engineering use, serving more as a proof of concept than a predictive tool.

Summary updated

Abstract

Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the *collapsing domain problem*, which can arise in architectures that avoid exploding gradients. ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks. We show this is a direct consequence of the Pythagorean equation. By noticing that *any neural network is a residual network*, we devise the *residual trick*, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.
BibTeX
@inproceedings{philipp2018gradients,
  author = {George Philipp and Dawn Song and Jaime G. Carbonell},
  title = {{Gradients explode - Deep Networks are shallow - ResNet explained}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
  url = {https://openreview.net/forum?id=J0zRlARSq8},
}
Figure