REPAIR: REnormalizing Permuted Activations for Interpolation Repair

Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, Behnam Neyshabur

When we train two neural networks from different random initializations on the same data, they usually end up in different "valleys" of the loss landscape. If you try to average their weights directly, the resulting model typically performs no better than random chance. This performance dip is known as the interpolation barrier.

In their ICLR 2023 paper, "REPAIR: REnormalizing Permuted Activations for Interpolation Repair," Keller Jordan and colleagues investigate why this barrier exists and provide a surprisingly simple fix that allows us to merge deep networks more effectively than ever before.

TL;DR

The authors identify "variance collapse"—a systematic decrease in activation signal strength—as the primary reason why interpolated neural networks fail. They introduce REPAIR, a method that recalibrates these activation statistics, reducing the interpolation barrier by up to 90% and enabling the "constructive merging" of models trained on separate data.

What Problem Does This Paper Address?

Recent research into Linear Mode Connectivity (LMC) suggests that if we account for the fact that neurons can be permuted (reordered) without changing a network's function, we might find a linear path of high accuracy between any two solutions.

However, even after aligning neurons so that "Feature A" in Model 1 matches "Feature A" in Model 2, deep networks like ResNets still show a massive drop in accuracy when their weights are averaged. The authors set out to find the "hidden" obstacle that remains after neuron alignment. They discover a phenomenon they call variance collapse: as data flows through an averaged network, the variance of the activations progressively shrinks. By the final layers, the signal is so muffled that the network can no longer distinguish between different inputs.

Variance collapse across layers before and after REPAIR

Methods

The authors propose a two-step process to "repair" interpolated models:

  1. Neuron Alignment: First, use existing algorithms (like the Hungarian algorithm) to permute the hidden units of one model to match the other based on activation correlations.
  2. REPAIR (The Core Contribution): After weight interpolation, the authors apply a per-channel rescaling and shifting (an affine transformation) to the preactivations.
  3. They calculate the "target" mean and variance for each channel as a weighted average of the statistics from the two original endpoint models.
  4. They then force the interpolated model's activations to match these targets.
  5. Implementation Trick: This is elegantly implemented by temporarily inserting BatchNorm layers in "train" mode to measure the collapsed stats, then setting the BatchNorm's internal weights and biases to achieve the target stats, and finally fusing these parameters back into the preceding convolutional or linear layers.

Datasets

The researchers validated REPAIR across a diverse set of benchmarks:

  • Image Classification: MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100.
  • Large-Scale: ImageNet (using ResNet18 and ResNet50).

Key Insights and Ideas

  • The Statistical "Why": Variance collapse is a mathematical inevitability of averaging. If two neurons are not perfectly correlated, the variance of their average is strictly less than the average of their variances. In deep networks, this effect compounds layer by layer.
  • LayerNorm is a Natural Shield: The authors found that models using Layer Normalization (common in Transformers) are naturally more resistant to interpolation barriers. This is because LayerNorm renormalizes activations during every forward pass, preventing the variance from collapsing as it moves through the depth of the network.
  • Depth vs. Width: While increasing network width helps reduce the barrier, increasing depth significantly worsens it. REPAIR effectively "neutralizes" the negative impact of depth on interpolation.

Key Results

REPAIR achieves dramatic reductions in the loss barrier across various architectures:

Accuracy interpolation barrier reduction for ResNet50 on ImageNet
  • ResNet50 on ImageNet: Reduced the relative barrier by 74% (bringing midpoint accuracy from near-zero up to 56.5%).
  • ResNet18 on CIFAR-10: Reduced the barrier by 90%.
  • VGG11 on CIFAR-10: Showed that REPAIR works even for architectures without residual connections or internal normalization.
  • Constructive Merging: In a "split data" experiment on CIFAR-100, the authors showed that merging two models trained on different subsets of data using REPAIR resulted in a single model that outperformed both of its parents.

Limitations and Future Work

While REPAIR is a major step forward, it doesn't completely eliminate the barrier, especially on difficult tasks like ImageNet. The midpoint model still lags behind the endpoints in absolute accuracy.

Additionally, REPAIR requires a small "calibration" pass (about 5,000 examples) to estimate the activation statistics of the interpolated model. Future work could explore closed-form versions of REPAIR (which the authors touch upon in the appendix) that don't require any data passes, potentially making model merging instantaneous. This research opens the door for more efficient model ensembling and decentralized training where models are merged rather than trained from scratch.

Summary updated

Abstract

In this paper we empirically investigate the conjecture from Entezari et al. (2021) which states that if permutation invariance is taken into account, then there should be no loss barrier to the linear interpolation between SGD solutions. We conduct our investigation using standard computer vision architectures trained on CIFAR-10 and ImageNet. First, we observe a general phenomenon in which interpolated deep networks suffer a collapse in the variance of their activations. We demonstrate that an appropriate rescaling of the pre-activations of the interpolated networks ameliorates this problem and significantly reduces the barrier. Second, by combining this with an algorithm for finding permutations based on maximizing correlations between the activations of matched neurons, we are able to reduce the interpolation barrier for a standard ResNet18 trained on CIFAR-10 to 1.5% absolute test error. We explore the interaction between our method and the choice of normalization layer, and demonstrate its robustness across a variety of architectures and training sets.
BibTeX
@inproceedings{jordan2023repair,
  author = {Keller Jordan and Hanie Sedghi and Olga Saukh and Rahim Entezari and Behnam Neyshabur},
  title = {{REPAIR: REnormalizing Permuted Activations for Interpolation Repair}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=gU5sJ6ZggcX},
}
Figure