TL;DR
Researchers have discovered BUMP (Brief Unexpected Mid-interpolation Peak), a phenomenon where slightly mixing the weights of two aligned neural networks actually improves test accuracy before it eventually drops. By combining this weight-space mixing with traditional prediction-space mixing, they developed a new ensembling method called WPME that consistently outperforms standard techniques.
What Problem Does This Paper Address?
In the world of deep learning, we often talk about Linear Mode Connectivity (LMC). The idea is that if you take two independently trained neural networks and align their neurons (accounting for symmetries like permutations), you can draw a straight line between their weights in high-dimensional space without hitting a "barrier" of high loss. Usually, researchers focus on the midpoint of this line to see if the models are truly connected.
However, this paper looks closer at the endpoints. The authors noticed a strange, undocumented behavior: as you start moving from Model A toward Model B, the accuracy doesn't just stay flat or drop—it actually goes up briefly. This "BUMP" suggests that weight-space interpolation is doing something more interesting than just averaging; it’s acting as a unique form of model improvement that hasn't been systematically studied until now.
Methods
The researchers followed a rigorous pipeline to document and exploit this effect:
- Alignment: They used the "Git Re-Basin" algorithm to permute the neurons of one model to match another, ensuring they are in the same "basin" of the loss landscape.
- Weight Interpolation: They created new models by taking a weighted average of the parameters: $(1 - \lambda)\Theta_A + \lambda\Theta_B$. The BUMP typically occurs at very small values of $\lambda$ (e.g., $\lambda \approx 0.1$).
- Ablation Studies: To prove BUMP isn't just a trivial effect, they tested moving in random directions in weight space and adding noise to logits.
- WPME (Weight-Prediction Mixed Ensembling): They proposed a novel ensemble method that performs weight mixing first, then blends those results with the original model's predictions in logit space.
Datasets
The study primarily used standard computer vision benchmarks to ensure reproducibility and broad applicability:
- CIFAR-10: Used with ResNet-20 and a custom "Tiny-10" architecture.
- CIFAR-100: Used with ResNet-20 to test the effect on more complex label spaces.
Key Insights and Ideas
-
Alignment is the Catalyst: The BUMP is significantly amplified when models are permutation-aligned. In "naive" (unaligned) models, the effect is either absent or much weaker.
Figure 1. The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0. -
It’s Not Just Regularization: You might think mixing weights is just adding "smart noise" that regularizes the model. However, the authors showed that moving in random directions or adding Gaussian noise to the outputs does not replicate the BUMP. There is something specific about moving toward another trained, aligned model.
-
Robustness to Degradation: Surprisingly, the BUMP persists—and even gets stronger—under extreme conditions. It appears early in the training process (long before convergence) and remains even if you prune 95% of the network's weights.
Figure 3 demonstrates that the BUMP gain is actually higher at earlier training checkpoints than at full convergence. -
Weight Mixing $\neq$ Logit Mixing: While both improve performance, they do so differently. Weight mixing induces a distinct transformation of the predictive distribution that isn't captured by simply averaging the outputs (logits) of the two models.
Key Results
- Consistent Accuracy Gains: On CIFAR-10 with ResNet-20 (16x width), the BUMP provided a statistically significant test accuracy gain of approximately 0.18% to 0.23% just by weight mixing.
- WPME Superiority: The proposed Weight-Prediction Mixed Ensembling (WPME) outperformed both standard Logit ensembling and pure weight mixing. For a ResNet-20 at epoch 60, WPME achieved 92.61% accuracy, compared to 92.37% for logit mixing and 91.27% for weight mixing.
- Better Calibration: WPME doesn't just make the model more accurate; it makes it more "honest." It achieved lower Expected Calibration Error (ECE), meaning the model's confidence scores better reflect its actual likelihood of being correct.
Limitations and Future Work
The authors acknowledge that while they have empirically characterized the BUMP, the exact theoretical mechanism remains a hypothesis. They suggest that Model B provides an "informed regularization" that sharpens Model A's decision boundaries, but they haven't yet proven why this specific geometric interaction occurs.
Future work could involve exploring this effect in much larger models (like LLMs) or investigating if non-linear interpolation paths could yield even larger "BUMPs." For now, it serves as a simple, "free" way to squeeze extra performance out of merged models.
Summary updated