Mean Field Residual Networks: On the Edge of Chaos

Greg Yang, Samuel S. Schoenholz

Abstract

We study randomly initialized residual networks using mean field theory and the theory of difference equations. Classical feedforward neural networks, such as those with tanh activations, exhibit exponential behavior on the average when propagating inputs forward or gradients backward. The exponential forward dynamics causes rapid collapsing of the input space geometry, while the exponential backward dynamics causes drastic vanishing or exploding gradients. We show, in contrast, that by adding skip connections, the network will, depending on the nonlinearity, adopt subexponential forward and backward dynamics, and in many cases in fact polynomial. The exponents of these polynomials are obtained through analytic methods and proved and verified empirically to be correct. In terms of the "edge of chaos" hypothesis, these subexponential and polynomial laws allow residual networks to "hover over the boundary between stability and chaos," thus preserving the geometry of the input space and the gradient information flow. In our experiments, for each activation function we study here, we initialize residual networks with different hyperparameters and train them on MNIST. Remarkably, our initialization time theory can accurately predict test time performance of these networks, by tracking either the expected amount of gradient explosion or the expected squared distance between the images of two input vectors. Importantly, we show, theoretically as well as empirically, that common initializations such as the Xavier or the He schemes are not optimal for residual networks, because the optimal initialization variances depend on the depth. Finally, we have made mathematical contributions by deriving several new identities for the kernels of powers of ReLU functions by relating them to the zeroth Bessel function of the second kind.

The Depth Dilemma and the Edge of Chaos

In deep learning, increasing network depth is a double-edged sword. As signals propagate through many layers of a standard feedforward network, they tend to either collapse (vanishing gradients/stability) or explode (exploding gradients/chaos). Effective computation is hypothesized to occur only at the "edge of chaos"—a narrow regime where the network is neither too stable (forgetting the input) nor too chaotic (treating similar inputs as completely different).

While Residual Networks (ResNets) are known to work well in practice, the theoretical underpinnings of their robustness have remained elusive. This paper characterizes the average behavior of randomly initialized ResNets to explain their superior trainability and expressivity compared to vanilla architectures.

Theoretical Framework: Mean Field Theory for ResNets

The authors employ Mean Field Theory and the theory of difference equations to analyze signal propagation. By treating weights and biases as random variables, they track the evolution of activation statistics—specifically their lengths and correlations—through the layers. They analyze two idealized architectures: the Reduced Residual Network (RRN) and the Full Residual Network (FRN).

Evolution of input correlations $\epsilon^{(l)}$ across layers, demonstrating how ResNets maintain signal geometry by converging slowly to stable fixed points rather than collapsing exponentially.

By deriving recurrence relations for quantities like the cosine distance between inputs ($\epsilon$) and the gradient magnitude ($\chi$), the study compares ResNets against "vanilla" networks across different activation functions, such as tanh and $\alpha$-ReLU.

From Exponential to Polynomial Dynamics

A central finding is that skip connections fundamentally alter the internal dynamics of the network. In vanilla networks, signal geometry collapses exponentially. In ResNets, skip connections force the dynamics to become subexponential or polynomial. This "slows down" the transition into chaos or stability, effectively allowing the network to stay in the computational "sweet spot" for hundreds of layers.

Subexponential growth of gradients in Tanh ResNets, following $\exp(A\sqrt{l})$ rather than the rapid exponential growth seen in vanilla networks.

While vanilla tanh networks see gradients vanish or explode at a rate of $e^L$, tanh ResNets exhibit growth at a much gentler rate of $e^{\sqrt{L}}$. This preservation of gradient information flow is critical for training extremely deep models.

Key Insights: Tanh vs. ReLU

The authors discovered a fascinating dichotomy in what limits performance based on the activation function:

  • Tanh ResNets are primarily limited by trainability: Their performance is best predicted by the extent of gradient explosion ($\chi$).
  • ReLU ResNets are limited by expressivity: Their performance is predicted by the network's ability to distinguish between different inputs (the metric distance $s$).

Furthermore, the study proves that standard initialization schemes like He or Xavier are non-optimal for ResNets. These schemes assume the optimal variance is independent of depth, whereas the authors show that for ResNets, the optimal initialization variance must depend on the total depth of the network.

Empirical Validation and Mathematical Contributions

The theoretical predictions were verified using MNIST and synthetic Gaussian data. Remarkably, the "initialization time" theory accurately predicted test-time performance; by analyzing a network's random state before training, the authors could determine its eventual success.

Beyond architectural insights, the paper contributes new mathematical identities relating ReLU kernels to the zeroth Bessel function of the second kind, providing a rigorous foundation for analyzing rectified units in a mean-field context.

Conclusion and Future Work

This work provides a robust theoretical explanation for the success of ResNets, framing them as systems that "hover" on the edge of chaos. Future research will aim to incorporate the effects of Batch Normalization and extend this analysis to Convolutional and Attention-based architectures, which are staples of modern deep learning.

Summary updated

Abstract

We study randomly initialized residual networks using mean field theory and the theory of difference equations. Classical feedforward neural networks, such as those with tanh activations, exhibit exponential behavior on the average when propagating inputs forward or gradients backward. The exponential forward dynamics causes rapid collapsing of the input space geometry, while the exponential backward dynamics causes drastic vanishing or exploding gradients. We show, in contrast, that by adding skip connections, the network will, depending on the nonlinearity, adopt subexponential forward and backward dynamics, and in many cases in fact polynomial. The exponents of these polynomials are obtained through analytic methods and proved and verified empirically to be correct. In terms of the "edge of chaos" hypothesis, these subexponential and polynomial laws allow residual networks to "hover over the boundary between stability and chaos," thus preserving the geometry of the input space and the gradient information flow. In our experiments, for each activation function we study here, we initialize residual networks with different hyperparameters and train them on MNIST. Remarkably, our initialization time theory can accurately predict test time performance of these networks, by tracking either the expected amount of gradient explosion or the expected squared distance between the images of two input vectors. Importantly, we show, theoretically as well as empirically, that common initializations such as the Xavier or the He schemes are not optimal for residual networks, because the optimal initialization variances depend on the depth. Finally, we have made mathematical contributions by deriving several new identities for the kernels of powers of ReLU functions by relating them to the zeroth Bessel function of the second kind.
BibTeX
@inproceedings{yang2017mean,
  author = {Greg Yang and Samuel S. Schoenholz},
  title = {{Mean Field Residual Networks: On the Edge of Chaos}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=vt27PBagpx4},
}
Figure