Techniques that combine multiple sets of neural network weights—such as Stochastic Weight Averaging (SWA), Exponential Moving Averages (EMA), and model merging—have become essential tools for improving generalization and stabilizing training. By averaging parameters across different training steps or independent runs, these methods find wider, more robust optima in the loss landscape.
However, weight averaging introduces a subtle but destructive side effect: it breaks Batch Normalization (BN). When you average the parameters of two or more networks, the resulting model often performs poorly or diverges entirely until its internal normalization statistics are explicitly recalculated.
Understanding why this mismatch occurs, and how a simple post-training calibration pass resolves it, is key to successfully combining neural network parameters.
The Parameter-Statistic Misalignment Problem
Batch Normalization layers operate by tracking the running mean and variance of feature activations across the training dataset. These tracking statistics are tightly coupled to the specific weight values at the moment the data is processed.
When we apply weight averaging techniques like SWA, we compute a new set of parameters that represents a point in the loss landscape the network never actually visited during standard training forward passes. As Pavel Izmailov et al. observed, this creates an immediate mismatch. The stored running statistics in the BN layers remain anchored to the historical training trajectory, rendering them completely stale relative to the newly averaged weights.
This decoupling imposes a strict functional bottleneck. As documented by Daniel Morales-Brotons et al., the performance of averaged models (such as those using long-window EMAs) is frequently bottlenecked not by the quality of the weights themselves, but by the incompatibility of the stale BN statistics. Without a correction step, the model's internal layers receive activations with unexpected scales and shifts, leading to catastrophic performance degradation.
Why Non-Linearity Destroys Stored Statistics
To build intuition for why this happens, consider the mathematical nature of deep networks. A neural network is a highly non-linear composition of functions. If weight configuration $W_A$ produces an activation distribution with mean $\mu_A$, and weight configuration $W_B$ produces a distribution with mean $\mu_B$, setting the weights to the exact midpoint $W_{avg} = \frac{W_A + W_B}{2}$ will not produce an activation mean of $\frac{\mu_A + \mu_B}{2}$.
Because activations drift non-linearly with changes in parameters, we cannot simply average or copy historical BN statistics. In his work on Latest Weight Averaging (LAWA), Jean Kaddour demonstrated that attempting to copy statistics from previous checkpoints fails to resolve the underlying misalignment. Similarly, in distributed settings where parameters are synchronized via stochastic shuffling, Louis Fournier et al. noted that overlooking the explicit synchronization or recomputation of batch normalization statistics consistently results in suboptimal final models.
Weights: W_A ─────────────────► W_avg (Averaged) ◄───────────────── W_B
│
Activations: Act_A (Mean μ_A) ▼ Act_B (Mean μ_B)
True Mean ≠ (μ_A+μ_B)/2
[Requires Recalibration Pass]
The Solution: The Post-Averaging Calibration Pass
Fortunately, the fix is computationally inexpensive and conceptually straightforward: a dedicated recalibration pass.
Once the weights are averaged or fused, they are frozen. The model is then placed back into a forward-pass state where the running statistics of the BN layers are allowed to update, but backpropagation is disabled. By passing a portion of the training data through the network, the BN layers directly observe the true empirical distributions of the activations generated by the new weight state.
This procedure resets and realigns the running means and variances, perfectly adapting the normalization layers to the new parameter configuration. Vipul Gupta et al. leveraged this exact mechanism in parallelized large-batch training (SWAP), executing a dedicated forward pass immediately following weight fusion to stabilize the model before inference.
This calibration signal is remarkably robust. For example, in adversarial training, computing forward passes on adversarially perturbed images is incredibly expensive. However, Joong-Won Hwang et al. demonstrated that when weights are averaged across training steps, running the recalibration pass using cheap, natural images provides a completely sufficient signal to realign the normalization layers, bypassing the need for costly adversarial calibration.
Generalizing to Model Merging and Interpolation
The necessity of post-averaging normalization updates extends far beyond temporal weight averaging; it is a universal requirement whenever independent model states are combined.
When merging models trained on entirely different datasets, the internal activation scales become severely disrupted. Masanori Yamada et al. showed that recalculating BN statistics on a tiny surrogate dataset—even as few as 10 samples per class—is enough to restore internal activation scales and stabilize the merged model. In the context of continual learning, where weights are interpolated to balance stability and plasticity, Jan Wasilewski et al. showed that updating internal normalization statistics is a necessary repair step to prevent catastrophic forgetting caused by activation drift.
Taking this concept to its logical conclusion, Keller Jordan et al. introduced the REPAIR framework to address the "interpolation barrier" between completely different random initializations.
Instead of just resetting standard BN layers, the authors temporarily insert specialized normalization layers into the merged model to explicitly measure activation drift across the interpolation path. Once the required scale and shift corrections are calculated, they are mathematically fused directly into the weights of the preceding linear or convolutional layers. This permanently repairs the activation distributions without adding any computational overhead during inference.
Variance Shifts Beyond Weight Manipulation
The core insight behind this line of research—that a mismatch between actual activation distributions and stored normalization statistics degrades performance—also explains other long-standing deep learning anomalies.
A prime example is the historical incompatibility between Dropout and Batch Normalization. Shuo Chen et al. demonstrated that this conflict stems from a fundamental "variance shift." During training, dropout randomly zeroes out channels, which artificially inflates the variance of the remaining activations. When dropout is turned off during inference, the actual variance of the activations shifts dramatically, rendering the BN statistics recorded during training incorrect.
Just as with weight averaging, the authors showed that this disharmony can be resolved without retraining. By running a post-training calibration pass over the training data with dropout disabled, the moving averages are forced to align with the true inference-time activation distributions, recovering the lost performance.
Key Foundational Work
Averaging Weights Leads to Wider Optima and Better Generalization (2018) — contributed 2 concept elements to this direction.
Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift (2019) — contributed 2 concept elements to this direction.