When initializing a dense neural network, standard practice relies on foundational schemes like Xavier or Kaiming initialization to keep the variance of activations stable across layers. These methods operate under a crucial assumption: every neuron connects uniformly to every neuron in the next layer, meaning the "fan-in" (the number of incoming connections per neuron) is constant and predictable.
However, as highlighted by Nils Koster et al. in Signing the Supermask: Keep, Hide, Invert and by Haoyu Ma in Training Your Sparse Neural Network Better with Any Mask, applying these dense heuristics to sparse networks breaks the underlying mathematics. When a network is pruned or designed with sparse connectivity, the number of active connections drops significantly. If the weights retain their original dense-centric scales, the signal variance shrinks with every layer it passes through. By the time the forward pass reaches the final layers, the activations attenuate to near zero, causing gradients to vanish during backpropagation. Conversely, improper scaling can cause gradients to explode, rendering the network entirely untrainable from scratch.
The Mechanics of Variance Preservation
To restore trainability, a sparse network must be brought back to a stable regime—often referred to as the "edge of chaos"—where signals can propagate freely without degrading. The fundamental solution is sparsity-aware weight rescaling: adjusting the magnitude of the surviving weights to compensate for the missing connections.
Consider a layer where a fraction $p$ of the weights are kept (the "keep ratio") and $1-p$ are pruned. If we assume the inputs and weights are zero-mean independent random variables, the variance of the layer's output is directly proportional to the number of active weights. To counteract the $p$-fold reduction in variance, the remaining weights must be scaled up. Multiplying the surviving weights by $1/\sqrt{p}$ restores the layer's output variance to exactly what it would be in a dense configuration.
This simple yet powerful variance-matching principle forms the theoretical backbone of early pruning methods. Soufiane Hayou et al. demonstrated in Robust Pruning at Initialization that post-pruning weight rescaling prevents layer collapse and ensures stable gradient flow even at extreme sparsities up to 99.9%. Providing further theoretical justification, Hongru Yang et al. utilized Neural Tangent Kernel (NTK) analysis in On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks to prove that a randomly pruned network becomes asymptotically equivalent to its dense counterpart during training, provided the surviving weights are rescaled by the inverse square root of the keep ratio.
This principle applies not only to pruning after initialization but also to finding subnetworks within random weights. Vivek Ramanujan et al. showed in What's Hidden in a Randomly Weighted Neural Network? that when searching for a high-performing "hidden subnetwork" inside a frozen random network, adjusting the initial weight distribution's standard deviation based on the target subnetwork sparsity is essential to maintain consistent signal variance across layers.
Correcting for Heterogeneous Connectivity
While global rescaling works well when sparsity is uniformly distributed, real-world sparse networks often exhibit structural irregularities. In unstructured sparse networks, different layers may have different pruning rates, and individual neurons within the same layer can end up with wildly varying numbers of incoming connections.
Haoyu Ma observed that traditional initialization schemes fail when layer-wise densities vary, leading to ineffective initial gradient updates. To solve this, Haoyu Ma proposed using layer-wise weight rescaling or learning scalar coefficients to dynamically adjust the weights based on the specific mask connectivity.
At an even finer granularity, Utku Evci et al. identified in Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win that the primary obstacle to training sparse networks from scratch is "heterogeneous connectivity." When individual neurons have different fan-ins, a global or layer-wide average scaling factor is insufficient; the signal variance drifts locally, disrupting gradient flow. The authors showed that optimization can be significantly improved by scaling the weight variance for each neuron individually based on its actual number of incoming connections, rather than relying on a global average fan-in assumption.
Extending the Principle to Model Merging and Architecture Design
The intuition of expectation- and variance-preservation extends far beyond standard network pruning; it has become a vital tool for compressing large language models and stabilizing novel sparse architectures.
Delta Sparsification and Model Merging
When merging multiple fine-tuned language models, researchers often drop a massive percentage of the task-specific "delta parameters" (the weight changes relative to the base model) to prevent parameter interference. To do this without destroying performance, Le Yu et al. introduced DARE (Drop And REscale) in Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, which uses expectation-preserving weight rescaling to multiply the remaining delta updates by a factor accounting for the drop rate. Similarly, Yan Yang et al. developed ImPart in ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs, an importance-aware framework that sparsifies delta parameters via Singular Value Decomposition (SVD) and rescales the remaining weights inversely proportional to the keep rate of their specific singular vectors, preventing shifts in internal hidden states.
Structured Sparse Architectures
When designing specialized sparse layers from scratch, variance matching is critical to ensure smooth integration with dense components. For instance, Keivan Alizadeh-Vahid et al. designed a specialized initialization for the Butterfly Transform: An Efficient FFT Based Neural Architecture Design to stabilize signal variance across recursive chains of sparse multiplications. For ultra-sparse networks operating at just 1% connectivity, Yingtao Zhang et al. derived a dedicated initialization in Epitopological learning and Cannistraci-Hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning to preserve signal propagation and prevent degradation.
Finally, in scaling massive memory networks up to 120B parameters, Zihao Huang et al. demonstrated in UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning that calibrating the initialization of sparse memory layers so that their output variance perfectly matches that of standard dense feed-forward networks eliminates the training divergence that typically plagues large-scale heterogeneous architectures.
Key Foundational Work
Robust Pruning at Initialization (2021) — contributed 3 concept elements to this direction.
What's Hidden in a Randomly Weighted Neural Network? (2020) — contributed 1 concept elements to this direction.
Butterfly Transform: An Efficient FFT Based Neural Architecture Design (2020) — contributed 1 concept elements to this direction.