In the early days of deep learning, training a network with more than a few layers was notoriously difficult. Signals would either vanish into numerical insignificance or explode into infinity before reaching the final layer. The breakthrough that enabled the modern era of deep architectures was the realization that the "health" of a neural network depends almost entirely on how its weights are first set.
The core objective of modern initialization is variance preservation: ensuring that the statistical distribution of activations and gradients remains stable as they propagate through dozens or hundreds of layers.
The Stability Problem
When a signal passes through a linear layer, it is multiplied by a weight matrix. If the weights are too small, the variance of the signal shrinks at every layer, eventually collapsing to zero. If the weights are too large, the variance grows exponentially.
Xavier Glorot and Yoshua Bengio famously diagnosed this in Understanding the difficulty of training deep feedforward neural networks. They proposed that to keep the signal stable, the variance of the weights should be scaled inversely by the number of input and output units (the "fan-in" and "fan-out"). This ensures that the forward-pass activations and backward-pass gradients have roughly the same magnitude across the entire network.
Generalizing to Any Activation
While "Xavier" and "He" initialization became standard for Tanh and ReLU activations, modern research has moved toward a unified theory. The framework VI3NR demonstrates that these classic techniques are actually special cases of a more general principle. By deriving weight variance based on the specific mean and variance of any chosen activation function—whether it is a standard ReLU or an exotic periodic function like a Sinc—we can maintain stable signal flow.
This is particularly critical for architectures with learnable or non-standard activations:
- Learnable Basis Functions: Ismail Khalfaoui-Hassani shows in Learnable polynomial, trigonometric, and tropical activations that calculating specific forward and backward gains for learnable coefficients eliminates the need for heuristic clamping or heavy normalization.
- Periodic Activations: For Sinusoidal Representation Networks (SIRENs), Antoine Venaille and Nelly Pustelnik developed a closed-form solution using the Lambert W function in A new initialisation to Control Gradients in Sinusoidal Neural network. This prevents high-frequency artifacts by keeping the network in a "quasi-linear" regime during early training.
- Maxout Networks: Hanna Tseran and Guido Montúfar found that for competitive activations, the variance must be scaled inversely with the second moment of the largest order statistic of the inputs, as detailed in Expected Gradients of Maxout Networks.
The Geometry of the Goldilocks Zone
Why does this statistical balancing act work so well? Adam Scherlis provides a geometric intuition in The Goldilocks zone. Weight initialization functions as a mechanism to control the starting radius of a model in parameter space. By tuning the variance, we ensure the model begins training within a specific "spherical shell" where the loss landscape is highly convex and conducive to optimization.
If we stray from this zone, the network may enter a "chaotic phase." While chaos is usually avoided, Initializing ReLU networks in an expressive subspace of weights suggests that carefully controlled chaos—achieved through anti-correlated weights—can actually prevent deep networks from becoming too "correlated," where distinct inputs become indistinguishable.
Beyond the Second Moment
Most initialization schemes focus on the second moment (variance). However, Mert Gurbuzbalaban and Yuanhan Hu argue in Fractional moment-preserving initialization schemes that this may be insufficient. By preserving fractional moments (order $s < 2$), researchers can account for the heavy-tailed weight distributions often seen in real training. This approach provides a "warm start" that is more robust against the "dying neuron" problem in ReLU networks.
Architecture Specifics: Transformers and Operators
As architectures grow more complex, variance preservation must be adapted to specific structural motifs:
- Transformers: Liyuan Liu and Jianfeng Gao identified the "amplification effect" in deep Transformers, where residual branches overwhelm the signal. Their Admin method, introduced in Understanding the Difficulty of Training Transformers, uses a profiling pass to set initialization constants that bound this change.
- State Space Models: In How to Train Your HiPPO, Albert Gu shows that projection matrices must maintain unit variance to ensure numerical stability when decomposing signals into orthogonal polynomials.
- Self-Normalizing Networks: Günter Klambauer took this to the logical extreme with Self-Normalizing Neural Networks, using the SELU activation and a specific variance-scaled initialization to ensure that activations automatically converge to zero mean and unit variance without explicit normalization layers.
By treating initialization not as a random starting point, but as a principled method for signal preservation, researchers have turned the "black art" of training deep networks into a rigorous branch of statistical engineering.