Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

Wojciech Tarnowski, Piotr Warchol, Stanislaw Jastrzebski, Jacek Tabor, Maciej A. Nowak

Abstract

We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespective of the activation function used. We do that by deriving, with the help of Free Probability, an analytical formula for the singular value distribution of the Jacobian, showing it depends on a single universal parameter.

Introduction

In the quest to train ever-deeper neural networks, researchers have long battled the "vanishing and exploding gradient" problem. One of the most powerful theoretical tools to emerge is Dynamical Isometry—the idea that if the input-output Jacobian of a network has singular values near 1, signals can propagate through thousands of layers without distortion.

Previously, achieving this was thought to require specific combinations of orthogonal weight initializations and antisymmetric activation functions like tanh. However, recent research demonstrates that for Residual Networks (ResNets), dynamical isometry is a "universal" property achievable with almost any activation function, provided the initialization is scaled correctly.

The Universal Jacobian Spectrum

Using Random Matrix Theory (RMT) and Free Probability Theory (FPT), the authors model the input-output Jacobian of a deep ResNet as a product of random matrices. They show that the entire spectrum of the Jacobian in the limit of infinite depth and width is determined by a single "effective cumulant" ($c$). This parameter $c$ is a function of the weight variance, the bias variance, and the average squared derivative of the chosen activation function.

Unlike vanilla feedforward networks, ResNets are remarkably robust. As long as the weight initialization variance is tuned to keep the parameter $c$ constant, the gradient flow (the Jacobian spectrum) remains identical regardless of whether one uses ReLU, Sigmoid, or Tanh.

Figure 1: Empirical singular value spectra for various activation functions (ReLU, Tanh, Sigmoid, SELU, linear) collapse onto the same theoretical curve when the effective cumulant c is fixed.

Empirical Training Performance

This theoretical universality has direct practical implications. On benchmarks like CIFAR-10, networks using wildly different activations show nearly identical training accuracy curves in the initial phases of learning, provided they are initialized to the same "universal" isometry point. This suggests that many perceived performance differences between activation functions in literature may actually be artifacts of sub-optimal initialization rather than inherent properties of the functions themselves.

Figure 2: Training accuracy on CIFAR-10. The plots demonstrate that disparate activation functions achieve parity in training speed when properly initialized at the point of dynamical isometry.

Conclusion

ResNets naturally concentrate their singular values around 1 much more easily than feedforward networks. By applying the $1/LN$ scaling law for weight variance, practitioners can "level the playing field," ensuring stable gradient flow across a vast range of architectural choices. This work shifts the focus from choosing the "right" activation function to correctly parameterizing the network's initialization to reach the universal regime of dynamical isometry.

Summary updated

Abstract

We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespective of the activation function used. We do that by deriving, with the help of Free Probability an...
BibTeX
@inproceedings{tarnowski2019dynamical,
  author = {Wojciech Tarnowski and Piotr Warchol and Stanislaw Jastrzebski and Jacek Tabor and Maciej A. Nowak},
  title = {{Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2019},
  url = {https://openreview.net/forum?id=zj5mbTvSMc2},
}
Figure