Weight Norm, Not Fan-in: Rethinking Normalization-Induced Gradient Skew in Sparse Training

Sparse neural networks keep only a small fraction of their connections, and dynamic sparse training also keeps rewiring which connections exist. When such a network uses batch normalization (a layer that rescales each neuron's output to zero mean and unit variance over the mini-batch), neurons with fewer surviving incoming weights get disproportionately large gradients. SparseOpt derived this distortion, which it calls gradient skew, as a factor of $(1-s_i)^{-1/2}$, where $s_i$ is the fraction of neuron $i$'s incoming weights that have been pruned. It cancels the skew with a per-neuron rescaling of the gradient, $\sqrt{1-s_i}$. That fix halves ImageNet training time. It also leaves an unexplained puzzle: the uncorrected, skewed gradients are better for choosing which connections to regrow.

We propose that the skew is not really about sparsity, and not about shifting activation distributions either. It is about the length of each neuron's incoming weight vector. A neuron followed by batch normalization produces the same output whatever the length of its weight vector, so its gradient scales as one over that length. Pruning shortens the vector, and this is why sparsity appears in the formula. During training, however, weight decay, learning, and regrowth of connections at zero all change weight length separately from fan-in (the count of active incoming weights). If this reading is right, there are three consequences. A preconditioner computed from the measured weight norm should beat one computed from sparsity, and it costs nothing extra to compute. The regrowth puzzle should turn out to be an implicit preference for growing connections into short-norm neurons, which can be tuned explicitly. And the batch-normalization correction should not carry over to the places in pre-normalization Transformers where no per-neuron scale invariance exists.

Core Claim

The internal-covariate-shift literature has already been through this kind of correction once. Batch Normalization attributed its speedup to stabilizing layer input distributions. How Does Batch Normalization Help Optimization? then showed that networks with deliberately unstable distributions after batch normalization train just as fast, and traced the benefit to gradient smoothness and scale. CDF Normalization found that removing distribution shift completely actually hurts. We take the same lesson into sparse training. The sparsity-and-normalization papers describe the problem in distributional language: heterogeneous fan-in breaks the normalizer's statistical assumptions. We argue it is better described in geometric terms.

For a weight vector $w_i$ whose output passes through batch normalization, the loss is unchanged under rescaling $w_i \to c\,w_i$. It follows that

$$\nabla_{w_i}\mathcal{L} = \frac{1}{\|w_i\|}\,P_i\,g_i, \qquad \text{effective angular step} \approx \frac{\eta}{\|w_i\|^2}\,\|P_i g_i\|,$$

where $P_i$ projects out the direction of $w_i$, $g_i$ is the gradient with respect to the unit-normalized weights, and $\eta$ is the learning rate. When per-weight magnitudes are roughly equal, as at initialization, $\|w_i\| \propto \sqrt{1-s_i}$, and this recovers SparseOpt's factor exactly. Our proposed correction replaces sparsity with measured norm:

$$D_{ii} = \|w_i\| \,/\, \bar r,$$

where $\bar r$ is the mean norm in the layer, kept so that the overall learning rate stays comparable.

The claim is falsifiable in a simple way. If $\|w_i\|$ and $\sqrt{1-s_i}$ stay tightly correlated throughout training, the norm-based preconditioner reduces to SparseOpt and the idea fails. We expect them to diverge for three reasons. Regrown weights in methods like RigL start at zero. Weight decay pulls norms toward an equilibrium that does not depend on fan-in. And neurons that gain or lose connections carry norms inherited from their history.

Research Program

The first part is measurement. In RigL and SET training of ResNet-20 and ResNet-18 on CIFAR-100 and ResNet-50 on ImageNet, we log per-neuron $s_i$, $\|w_i\|$, and the actual gradient norm at every rewiring step. We then check which of $1/\|w_i\|$ and $(1-s_i)^{-1/2}$ better predicts the measured gradient amplification. This tells us whether the mechanism matters before any method is built.

The second part is the method. Norm-calibrated preconditioning applies $D_{ii}$ to weight updates, computed from the current weights at each step. Because it is computed fresh every step, it adjusts the moment a rewiring changes a neuron. That matters because the abrupt topology jumps that SparseOpt identifies as shocks are exactly where running statistics go stale. We also implement the constraint-based version of the same idea: projecting each sparse neuron's active weights onto a fixed-radius sphere, adapted from Constrained Weight Optimization. With equal norms there is nothing left to correct. The two implementations test the same mechanism.

The third part addresses the regrowth puzzle. RigL ranks candidate connections by the magnitude of their gradient at zero. Under our account, the raw score of a candidate into neuron $i$ carries a factor $1/\|w_i\|$, which is a built-in bonus for growing into short-norm, typically sparse neurons. We separate the two pieces by scoring candidates with the norm-corrected gradient times an explicit priority $\|w_i\|^{-\alpha}$, and sweeping $\alpha$. Skewed regrowth corresponds to $\alpha=1$ and corrected regrowth to $\alpha=0$. The prediction is that the best $\alpha$ is neither of these. A second prediction is that under sphere projection, where norms are equal, the skewed-versus-corrected difference in regrowth largely disappears. If it does not disappear, fan-in itself carries signal beyond norm, and we would report that.

The fourth part tests where the correction should transfer. In pre-normalization Transformers, sparse linear layers feed the residual stream, and RMSNorm (Root Mean Square Layer Normalization) normalizes the whole token vector rather than each neuron. So per-neuron scale invariance holds only where a per-neuron or per-head normalization sits directly after the sparse weights. Our account predicts that SparseOpt-style correction will help little, or even hurt, in ordinary pre-normalization layers, and will help where scale invariance is present. This gives a principled answer to the open question about LayerNorm and RMSNorm that Understanding Normalization Layers for Sparse Training left untested empirically. It also fits the finding in When Does Sparsity Mitigate the Curse of Depth in LLMs that sparsity there mostly changes residual variance rather than per-neuron gradient scale.

Baselines

The naive reference point is RigL with SGD and no preconditioning. This is the setting in which skew was first diagnosed. SparseOpt reports 69.94% ImageNet top-1 for ResNet-50 at 97% sparsity after 90 epochs. We rerun it with our own code and the same schedule, so that every comparison uses one implementation.

The strongest existing method is SparseOpt with RigL. It applies the $\sqrt{1-s_i}$ preconditioner to updates and keeps skewed gradients for regrowth, and it reports 71.15% top-1 in the same setting. On CIFAR-100 without skip connections at 80% sparsity, the closely related preconditioner in Understanding Normalization Layers for Sparse Training reports 69.21% against a 68.74% baseline. For extreme sparsity on CIFAR-100, Gradient-based Weight Density Balancing reports about 58% at 99% sparsity with uniform initialization, and we include it as the dynamic-reallocation competitor.

All methods share data, augmentation, epochs, rewiring schedule, and sparsity distribution. Each receives the same learning-rate and weight-decay sweep of equal size, and we average over three seeds. Beating SparseOpt means a top-1 gain larger than the seed standard deviation at 95% and 97% sparsity within 90 epochs, or matching its accuracy in noticeably fewer epochs.

The key ablation isolates the ingredient that matters by comparing preconditioners that differ only in their per-neuron statistic. The candidates are $\sqrt{1-s_i}$, $\|w_i\|$, and a shuffled $\|w_i\|$ that keeps the distribution of norms but breaks the link to each neuron. A second ablation crosses the update preconditioner with the regrowth scoring rule.

Why not just…?

Why not just use Adam, which already rescales each coordinate? Adam's second-moment estimate is a running average over time. A rewiring step changes a neuron's norm all at once, so the estimate lags behind, a weakness SparseOpt notes as well. Adam is also per-coordinate and does not single out the per-neuron scale direction. We run AdamW with the same tuning budget, with and without norm calibration. If plain AdamW matches our method, the per-step correction is unnecessary in adaptive training, and we will report that.

Why not just switch to LayerNorm or GroupNorm, or remove normalization altogether? The research file reports that LayerNorm shows the same amplification on sparse neurons (Understanding Normalization Layers for Sparse Training). Weight Standardization reports that GroupNorm without weight standardization suffers from dead units. Normalizer-free networks (Characterizing signal propagation) calibrate their variance-scaling constants for dense layers, and the depth paper shows that sparsity changes variance by 3 to 6 times. Weight standardization is a separate, real case. It explicitly divides by the weight vector's spread, which makes neurons scale-invariant by construction. Our account predicts that skew reappears there unless the statistics are computed over active weights only. We test this with a sparse-aware standardization variant.

Why not just initialize sparse neurons with fan-in-scaled weights so that norms start equal? This fixes the problem at step zero. Our account says norms drift away from fan-in afterwards. The measurement study shows directly whether this drift happens. We also include fan-in-aware initialization with an unchanged optimizer as a baseline. If it matches the preconditioners, the drift is too small to matter.

Why not just project weights onto a sphere, as in constrained weight optimization? This is a serious threat, because it is the same mechanism implemented as a constraint. We treat it as a second form of our method rather than as a rival. Whichever of the two wins, the norm hypothesis is supported, provided both beat the sparsity-based preconditioner. The practical choice between them then comes down to cost, and to whether projection interacts badly with regrowth at zero.

Proposed Experiments

The first decisive check is the measurement study. If the correlation between $\|w_i\|$ and $\sqrt{1-s_i}$ stays above roughly 0.95 throughout training on CIFAR-100 and ImageNet, we stop. The phenomenon is then fully captured by SparseOpt.

The second check is on CIFAR-100. We train ResNet-20 and ResNet-18 at 90%, 95%, 97%, and 99% sparsity, with and without skip connections, which reaches the residual setting Understanding Normalization Layers for Sparse Training omitted. We compare the update-preconditioner ablation and the regrowth sweep over $\alpha$, and we track final accuracy, epochs to a target accuracy, and the largest Hessian eigenvalue relative to $2/\eta$, the stability threshold.

The third check is ImageNet ResNet-50 at 95% and 97% sparsity for 90 epochs against the SparseOpt figures above. We run it only for the configuration selected on CIFAR, to keep the tuning budget fair.

The fourth check uses a small sparse pre-normalization Transformer, and a mixture-of-experts variant in which a learned gate sends each token to a few expert sub-networks. We train these on FineWeb-Edu at the low end of the scale studied in When Does Sparsity Mitigate the Curse of Depth in LLMs, and measure perplexity. We apply each correction once only where per-neuron scale invariance exists and once uniformly everywhere. The prediction is that uniform application gives no gain or a loss, while targeted application gives a gain. We have no reported result on Lacuna for sparsity-aware preconditioning on this benchmark, so we establish our own baseline here.