Abstract
We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit. The resulting singular value spectrum depends on a single parameter, which we calculate for a variety of popular activation functions, by analyzing the signal propagation in the artificial neural network. We corroborate our results with numerical simulations of both random matrices and ResNets applied to the CIFAR-10 classification problem. Moreover, we study the consequence of this universal behavior for the initial and late phases of the learning processes. We conclude by drawing attention to the simple fact, that initialization acts as a confounding factor between the choice of activation function and the rate of learning. We propose that in ResNets this can be resolved based on our results, by ensuring the same level of dynamical isometry at initialization.
Introduction
Training very deep neural networks is notoriously difficult due to the vanishing and exploding gradient problem. As signals pass through many layers, they can either shrink to zero or grow uncontrollably, making optimization nearly impossible. A known solution is to achieve dynamical isometry, where the "input-output Jacobian" (the matrix of all partial derivatives of the output with respect to the input) has singular values concentrated around 1.
In standard feedforward networks, achieving this is a "fine-tuning" nightmare, often requiring specific activation functions like tanh and specialized orthogonal weight initializations. Residual Networks (ResNets), however, seem to maintain trainable gradients across a wide variety of architectures. This paper provides a rigorous theoretical foundation for this success, demonstrating that ResNets achieve dynamical isometry in a universal way regardless of the activation function used.
Methods
The authors employ Free Probability Theory (FPT) and Random Matrix Theory (RMT) to analytically compute the distribution of singular values for the entire network's Jacobian in the limit of infinite width ($N$) and depth ($L$). The core of their approach is deriving a universal formula for the spectral density of the Jacobian, which depends on a single parameter, $c$, termed the effective cumulant. This parameter encapsulates the specific choice of activation function and the variance of the weights and biases.
A critical finding is the $1/LN$ scaling rule: for ResNets, the variance of the weights should be scaled by $1/(L \cdot N)$, where $L$ is the number of skip connections and $N$ is the layer width. This scaling prevents signal explosion as it accumulates through the residual branches, maintaining the stability of the forward and backward passes.
Key Results
The theoretical predictions for the singular value distribution of the Jacobian matched numerical simulations perfectly, even for networks as shallow as 10 layers. Unlike vanilla networks, which belong to different "universality classes" depending on their activation function, ResNets fall into a single class. This means that whether using ReLU, Tanh, or SELU, the "shape" of the gradient flow is fundamentally the same if initialized correctly.
Numerical experiments on CIFAR-10 confirmed that when different activation functions (ReLU, Leaky ReLU, Tanh, SELU) were initialized to have the same effective cumulant ($c=0.125$), their training curves became remarkably similar. This confirms that the Jacobian spectrum is a primary driver of early learning dynamics. Interestingly, while Sigmoid can theoretically achieve isometry, the theory explains its practical failure: it causes pre-activations to grow so fast that neurons saturate almost immediately.
Conclusion
This work highlights that initialization acts as a confounding factor between the choice of activation function and the rate of learning. By ensuring the same level of dynamical isometry (the same $c$ value), researchers can finally compare activation functions on a level playing field. While the study focuses on fully connected ResNets and does not fully integrate the complexities of Batch Normalization or spatial convolutional structures, it provides a powerful framework for understanding why ResNets are so robust and how to optimize their initialization for any activation function.
Summary updated