Abstract
We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespective of the activation function used. We do that by deriving, with the help of Free Probability, an analytical formula for the singular value distribution of the Jacobian, showing it depends on a single universal parameter.
Introduction
In the quest to train ever-deeper neural networks, researchers have long battled the "vanishing and exploding gradient" problem. One of the most powerful theoretical tools to emerge is Dynamical Isometry—the idea that if the input-output Jacobian of a network has singular values near 1, signals can propagate through thousands of layers without distortion.
Previously, achieving this was thought to require specific combinations of orthogonal weight initializations and antisymmetric activation functions like tanh. However, recent research demonstrates that for Residual Networks (ResNets), dynamical isometry is a "universal" property achievable with almost any activation function, provided the initialization is scaled correctly.
The Universal Jacobian Spectrum
Using Random Matrix Theory (RMT) and Free Probability Theory (FPT), the authors model the input-output Jacobian of a deep ResNet as a product of random matrices. They show that the entire spectrum of the Jacobian in the limit of infinite depth and width is determined by a single "effective cumulant" ($c$). This parameter $c$ is a function of the weight variance, the bias variance, and the average squared derivative of the chosen activation function.
Unlike vanilla feedforward networks, ResNets are remarkably robust. As long as the weight initialization variance is tuned to keep the parameter $c$ constant, the gradient flow (the Jacobian spectrum) remains identical regardless of whether one uses ReLU, Sigmoid, or Tanh.
Empirical Training Performance
This theoretical universality has direct practical implications. On benchmarks like CIFAR-10, networks using wildly different activations show nearly identical training accuracy curves in the initial phases of learning, provided they are initialized to the same "universal" isometry point. This suggests that many perceived performance differences between activation functions in literature may actually be artifacts of sub-optimal initialization rather than inherent properties of the functions themselves.
Conclusion
ResNets naturally concentrate their singular values around 1 much more easily than feedforward networks. By applying the $1/LN$ scaling law for weight variance, practitioners can "level the playing field," ensuring stable gradient flow across a vast range of architectural choices. This work shifts the focus from choosing the "right" activation function to correctly parameterizing the network's initialization to reach the universal regime of dynamical isometry.
Summary updated