BUMP: Brief Unexpected Mid-Interpolation Peak

Melinda Flóra Kiss, Márton Muntag, Adrián Csiszárik, Dániel VargaarXiv 2026 · arXiv

TL;DR

Researchers have discovered BUMP (Brief Unexpected Mid-interpolation Peak), a phenomenon where slightly mixing the weights of two aligned neural networks actually improves test accuracy before it eventually drops. By combining this weight-space mixing with traditional prediction-space mixing, they developed a new ensembling method called WPME that consistently outperforms standard techniques.

What Problem Does This Paper Address?

In the world of deep learning, we often talk about Linear Mode Connectivity (LMC). The idea is that if you take two independently trained neural networks and align their neurons (accounting for symmetries like permutations), you can draw a straight line between their weights in high-dimensional space without hitting a "barrier" of high loss. Usually, researchers focus on the midpoint of this line to see if the models are truly connected.

However, this paper looks closer at the endpoints. The authors noticed a strange, undocumented behavior: as you start moving from Model A toward Model B, the accuracy doesn't just stay flat or drop—it actually goes up briefly. This "BUMP" suggests that weight-space interpolation is doing something more interesting than just averaging; it’s acting as a unique form of model improvement that hasn't been systematically studied until now.

Methods

The researchers followed a rigorous pipeline to document and exploit this effect:

  1. Alignment: They used the "Git Re-Basin" algorithm to permute the neurons of one model to match another, ensuring they are in the same "basin" of the loss landscape.
  2. Weight Interpolation: They created new models by taking a weighted average of the parameters: $(1 - \lambda)\Theta_A + \lambda\Theta_B$. The BUMP typically occurs at very small values of $\lambda$ (e.g., $\lambda \approx 0.1$).
  3. Ablation Studies: To prove BUMP isn't just a trivial effect, they tested moving in random directions in weight space and adding noise to logits.
  4. WPME (Weight-Prediction Mixed Ensembling): They proposed a novel ensemble method that performs weight mixing first, then blends those results with the original model's predictions in logit space.

Datasets

The study primarily used standard computer vision benchmarks to ensure reproducibility and broad applicability:

  • CIFAR-10: Used with ResNet-20 and a custom "Tiny-10" architecture.
  • CIFAR-100: Used with ResNet-20 to test the effect on more complex label spaces.

Key Insights and Ideas

  • Alignment is the Catalyst: The BUMP is significantly amplified when models are permutation-aligned. In "naive" (unaligned) models, the effect is either absent or much weaker.

    Figure 1. The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.
  • It’s Not Just Regularization: You might think mixing weights is just adding "smart noise" that regularizes the model. However, the authors showed that moving in random directions or adding Gaussian noise to the outputs does not replicate the BUMP. There is something specific about moving toward another trained, aligned model.

  • Robustness to Degradation: Surprisingly, the BUMP persists—and even gets stronger—under extreme conditions. It appears early in the training process (long before convergence) and remains even if you prune 95% of the network's weights.

    Figure 3 demonstrates that the BUMP gain is actually higher at earlier training checkpoints than at full convergence.
  • Weight Mixing $\neq$ Logit Mixing: While both improve performance, they do so differently. Weight mixing induces a distinct transformation of the predictive distribution that isn't captured by simply averaging the outputs (logits) of the two models.

Key Results

  • Consistent Accuracy Gains: On CIFAR-10 with ResNet-20 (16x width), the BUMP provided a statistically significant test accuracy gain of approximately 0.18% to 0.23% just by weight mixing.
  • WPME Superiority: The proposed Weight-Prediction Mixed Ensembling (WPME) outperformed both standard Logit ensembling and pure weight mixing. For a ResNet-20 at epoch 60, WPME achieved 92.61% accuracy, compared to 92.37% for logit mixing and 91.27% for weight mixing.
  • Better Calibration: WPME doesn't just make the model more accurate; it makes it more "honest." It achieved lower Expected Calibration Error (ECE), meaning the model's confidence scores better reflect its actual likelihood of being correct.

Limitations and Future Work

The authors acknowledge that while they have empirically characterized the BUMP, the exact theoretical mechanism remains a hypothesis. They suggest that Model B provides an "informed regularization" that sharpens Model A's decision boundaries, but they haven't yet proven why this specific geometric interaction occurs.

Future work could involve exploring this effect in much larger models (like LLMs) or investigating if non-linear interpolation paths could yield even larger "BUMPs." For now, it serves as a simple, "free" way to squeeze extra performance out of merged models.

Summary updated

Abstract

We identify an intriguing phenomenon when linearly interpolating between permutation-aligned neural networks: test accuracy briefly improves near the interpolation endpoints before dropping toward the midpoint, which we term BUMP (Brief Unexpected Mid-interpolation Peak). We observe that BUMP appears consistently across datasets, architectures, training regimes, and remains robust under substantial perturbations such as early checkpoints and extreme pruning. Moreover, controlled ablations suggest that it cannot be attributed to trivial regularization effects. While similar near-endpoint gains are visible in prior interpolation plots, they have not been explicitly documented or systematically and thoroughly analyzed. Leveraging this effect, we propose a simple, novel ensembling scheme: weight-prediction mixed ensembling that combines weight mixing with prediction-space mixing. Across our experiments, the resulting ensembling method consistently outperforms standard logit ensembling in test accuracy and yields improved calibration trade-off, highlighting complementary benefits of weight- and prediction-space mixing.
BibTeX
@misc{kiss2026bump,
  author = {Melinda Flóra Kiss and Márton Muntag and Adrián Csiszárik and Dániel Varga},
  title = {{BUMP: Brief Unexpected Mid-Interpolation Peak}},
  year = {2026},
  eprint = {2611.08182},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08182},
}
Figure