When we train two neural networks from different random initializations on the same data, they usually end up in different "valleys" of the loss landscape. If you try to average their weights directly, the resulting model typically performs no better than random chance. This performance dip is known as the interpolation barrier.
In their ICLR 2023 paper, "REPAIR: REnormalizing Permuted Activations for Interpolation Repair," Keller Jordan and colleagues investigate why this barrier exists and provide a surprisingly simple fix that allows us to merge deep networks more effectively than ever before.
TL;DR
The authors identify "variance collapse"—a systematic decrease in activation signal strength—as the primary reason why interpolated neural networks fail. They introduce REPAIR, a method that recalibrates these activation statistics, reducing the interpolation barrier by up to 90% and enabling the "constructive merging" of models trained on separate data.
What Problem Does This Paper Address?
Recent research into Linear Mode Connectivity (LMC) suggests that if we account for the fact that neurons can be permuted (reordered) without changing a network's function, we might find a linear path of high accuracy between any two solutions.
However, even after aligning neurons so that "Feature A" in Model 1 matches "Feature A" in Model 2, deep networks like ResNets still show a massive drop in accuracy when their weights are averaged. The authors set out to find the "hidden" obstacle that remains after neuron alignment. They discover a phenomenon they call variance collapse: as data flows through an averaged network, the variance of the activations progressively shrinks. By the final layers, the signal is so muffled that the network can no longer distinguish between different inputs.
Methods
The authors propose a two-step process to "repair" interpolated models:
- Neuron Alignment: First, use existing algorithms (like the Hungarian algorithm) to permute the hidden units of one model to match the other based on activation correlations.
- REPAIR (The Core Contribution): After weight interpolation, the authors apply a per-channel rescaling and shifting (an affine transformation) to the preactivations.
- They calculate the "target" mean and variance for each channel as a weighted average of the statistics from the two original endpoint models.
- They then force the interpolated model's activations to match these targets.
- Implementation Trick: This is elegantly implemented by temporarily inserting BatchNorm layers in "train" mode to measure the collapsed stats, then setting the BatchNorm's internal weights and biases to achieve the target stats, and finally fusing these parameters back into the preceding convolutional or linear layers.
Datasets
The researchers validated REPAIR across a diverse set of benchmarks:
- Image Classification: MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100.
- Large-Scale: ImageNet (using ResNet18 and ResNet50).
Key Insights and Ideas
- The Statistical "Why": Variance collapse is a mathematical inevitability of averaging. If two neurons are not perfectly correlated, the variance of their average is strictly less than the average of their variances. In deep networks, this effect compounds layer by layer.
- LayerNorm is a Natural Shield: The authors found that models using Layer Normalization (common in Transformers) are naturally more resistant to interpolation barriers. This is because LayerNorm renormalizes activations during every forward pass, preventing the variance from collapsing as it moves through the depth of the network.
- Depth vs. Width: While increasing network width helps reduce the barrier, increasing depth significantly worsens it. REPAIR effectively "neutralizes" the negative impact of depth on interpolation.
Key Results
REPAIR achieves dramatic reductions in the loss barrier across various architectures:
- ResNet50 on ImageNet: Reduced the relative barrier by 74% (bringing midpoint accuracy from near-zero up to 56.5%).
- ResNet18 on CIFAR-10: Reduced the barrier by 90%.
- VGG11 on CIFAR-10: Showed that REPAIR works even for architectures without residual connections or internal normalization.
- Constructive Merging: In a "split data" experiment on CIFAR-100, the authors showed that merging two models trained on different subsets of data using REPAIR resulted in a single model that outperformed both of its parents.
Limitations and Future Work
While REPAIR is a major step forward, it doesn't completely eliminate the barrier, especially on difficult tasks like ImageNet. The midpoint model still lags behind the endpoints in absolute accuracy.
Additionally, REPAIR requires a small "calibration" pass (about 5,000 examples) to estimate the activation statistics of the interpolated model. Future work could explore closed-form versions of REPAIR (which the authors touch upon in the appendix) that don't require any data passes, potentially making model merging instantaneous. This research opens the door for more efficient model ensembling and decentralized training where models are merged rather than trained from scratch.
Summary updated