Variance Preserving Initialization for Deep Neural Networks

34 concepts33 papers1 proposal

Imagine passing a whisper down a line of one hundred people. If each person repeats the message slightly louder than they heard it, the final person will be met with a deafening scream. If each person speaks slightly quieter, the message will dissolve into absolute silence long before reaching the end.

This is the fundamental challenge of training deep neural networks. In a network with dozens or hundreds of layers, signals must travel forward during the activation phase and backward during the gradient propagation phase. Without careful calibration, the magnitude of these signals will either grow exponentially (exploding gradients) or shrink exponentially (vanishing gradients), rendering the network untrainable.

Variance-preserving initialization solves this by ensuring that the statistical distribution of activations and gradients remains remarkably stable from the very first step of training.

The Mathematics of Information Flow

To understand why variance-preserving initialization works, we can look at a single linear layer:

$$y = Wx$$

Here, $x$ is the input vector from the previous layer, $W$ is the weight matrix, and $y$ is the resulting pre-activation vector. Let's assume that both our inputs and weights are initialized randomly with a mean of zero, and that they are independent of one another.

When we compute a single element $y_i$, we are taking the dot product of one row of $W$ and the vector $x$:

$$y_i = \sum_{j=1}^{N_{\text{in}}} W_{ij} x_j$$

According to basic probability theory, the variance of a sum of independent random variables is the sum of their individual variances. Because every term in this sum shares the same distribution, the variance of the output scales linearly with the number of inputs ($N_{\text{in}}$, or "fan-in"):

$$\text{Var}(y_i) = N_{\text{in}} \cdot \text{Var}(W_{ij}) \cdot \text{Var}(x_j)$$

If we want the variance of our signal to remain unchanged as it passes through this layer—meaning $\text{Var}(y_i) = \text{Var}(x_j)$—the product $N_{\text{in}} \cdot \text{Var}(W_{ij})$ must equal exactly $1$. This reveals the foundational rule of weight scaling:

$$\text{Var}(W) = \frac{1}{N_{\text{in}}}$$

This elegant derivation forms the bedrock of modern deep learning optimization. As established in the seminal work of Glorot et al. (2010) and further detailed by Bengio (2012), balancing these variances across both the forward pass ($N_{\text{in}}$) and the backward pass ($N_{\text{out}}$) prevents numerical instability. As demonstrated by Du et al. (2019), scaling weight variances in this manner keeps the initial gradient signal highly informative, allowing optimization algorithms to escape complex, non-convex regions efficiently.

The Geometry of Activations

The linear equation breaks down the moment we introduce non-linear activation functions. Consider the Rectified Linear Unit (ReLU), which replaces all negative values with zero.

Because a symmetric distribution centered at zero will have exactly half of its values below zero, ReLU completely discards half of the incoming signal's energy. Consequently, passing a signal through a ReLU activation cuts its variance precisely in half.

If we continue to use the standard $1/N_{\text{in}}$ scaling, the signal variance will plummet by a factor of $2^L$ over $L$ layers. To counteract this geometric halving, He et al. (2015) proved that the initialization variance must be doubled:

$$\text{Var}(W) = \frac{2}{N_{\text{in}}}$$

Without this adjustment, deep networks face catastrophic failure modes before training even begins. As shown by Hanin et al. (2018), proper variance scaling is a strict prerequisite for deep ReLU networks to initiate training without immediate signal collapse.

This principle generalizes across activation geometries. For instance, Leaky ReLU functions do not zero out negative signals entirely; instead, they compress them by a slope parameter $\alpha$. To accommodate this, Tsuchida (2018) demonstrated that the initialization variance must be dynamically scaled based on $\alpha$ to preserve the expected variance across deep architectures.

Self-Normalization and Fixed Points

While standard initialization techniques aim to preserve variance at step zero, the statistics of the network can still drift during training. Can we design a system where the network naturally pulls its activations back toward a stable distribution?

This is the core intuition behind Self-Normalizing Neural Networks (SNNs). Klambauer et al. (2017) introduced the Scaled Exponential Linear Unit (SELU) and proved that when combined with a variance-scaled initialization ($\text{Var}(W) = 1/N_{\text{in}}$), the mapping from one layer's activations to the next forms a contraction mapping. This mapping possesses a stable fixed point at zero mean and unit variance. If the variance grows or shrinks slightly, the mathematical properties of the activation function force it back to $1.0$ in the subsequent layer. This property was later adapted by Madasu et al. (2019) to stabilize deep convolutional networks for text processing without requiring explicit normalization layers.

A similar fixed-point analysis is vital when dealing with structural constraints that break standard assumptions. For example, Input-Convex Neural Networks require non-negative weights, meaning the weight distribution cannot be zero-centered. To prevent exponential signal explosion in these architectures, Hoedt et al. (2023) derived a specialized initialization framework that identifies fixed points for activation statistics under non-centered weight distributions.

Generalizing to Exotic Architectures

As deep learning has expanded into implicit neural representations, hypercomplex spaces, and transformers, standard variance-preserving formulas have evolved to meet unique architectural demands.

Periodic and Learnable Activations

Implicit Neural Representations often utilize sinusoidal activations to capture fine details in images and 3D shapes. While early implementations required highly rigid, non-standard initialization recipes, Belbute-Peres et al. (2023) established that standard variance-preserving principles can successfully stabilize periodic networks. However, when the activation functions themselves are fully learnable—such as in the Kolmogorov-Arnold Transformer introduced by Yang et al. (2025)—specialized variance-preserving initializations become critical to prevent signal degradation across deep layers.

To unify these disparate solutions, the authors of VI^3NR (2025) developed a generalized initialization framework. By mathematically balancing the variance of pre-activations and gradients across all layers simultaneously, this approach ensures stable signal propagation for any arbitrary activation function.

High-Dimensional Algebras

When networks operate in hypercomplex domains, such as Deep Octonion Networks, standard Gaussian initialization fails because it ignores the unique dimensionality and non-associative multiplication rules of the underlying algebra. Xu et al. (2019) resolved this by scaling initializations using the Rayleigh distribution, mapping variance preservation directly to the geometric properties of 8-dimensional space.

Data-Driven and Geometric Refinements

Purely theoretical initialization formulas assume that the input data is perfectly independent and identically distributed. In practice, real-world data contains strong correlations that can skew activation variances.

To bridge the gap between theory and practice, Mishkin (2016) introduced Layer-sequential unit-variance (LSUV) initialization. LSUV decouples the process into two steps: first, it initializes weights with orthonormal matrices to preserve signal decorrelation; second, it passes a single minibatch of actual data through the network and empirically scales each layer's weights to ensure exact unit variance. A related data-dependent strategy was utilized by Salimans et al. (2016) within Weight Normalization, leveraging initial minibatch statistics to scale and bias parameters for a well-conditioned optimization start.

Finally, the geometric perspective offers alternatives to standard dimensional averages. Defazio (2022) proposed a scaling calculus demonstrating that initializing weight variance using the geometric mean of fan-in and fan-out—rather than the traditional arithmetic mean—provides a more balanced optimization landscape, maintaining a well-behaved Hessian matrix across varied datasets.

Key Foundational Work

Understanding the difficulty of training deep feedforward neural networks (2010) — contributed 1 concept elements to this direction.
Practical recommendations for gradient-based training of deep architectures (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_variance_preserving_initialization_13fdef8b,
  title = {variance-preserving initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-preserving-initialization/rdr_13fdef8b26f640bda3c8f27509d5cb70},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_variance_preserving_initialization_13fdef8b_v1,
  title = {variance-preserving initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-preserving-initialization/rdr_13fdef8b26f640bda3c8f27509d5cb70/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake