Suppose you train two identical neural network architectures on the exact same dataset, starting from different random seeds. Both models reach near-perfect accuracy. If you average their predictions—a standard ensembling technique—the resulting combination performs exceptionally well. But what happens if you average their parameters directly?
Intuitively, if both networks have found valid solutions to the same problem, their average parameter state might be expected to yield a reasonable intermediate solution. Instead, the resulting model typically performs no better than random chance.
In the optimization landscape of deep neural networks, this phenomenon is known as the linear mode connectivity (LMC) barrier or interpolation barrier. While high-performing solutions are often connected by complex, non-linear paths of low loss, a straight line drawn between them in parameter space almost always passes through a massive wall of high loss. Understanding why this barrier exists, when it disappears, and how to bypass it has become a central focus of research into the geometry of neural network loss landscapes.
The Geometry of the Barrier
To understand why naive weight averaging fails, we must look at the geometry of the loss surface. As shown by Timur Garipov et al. in Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, the loss landscape is not a collection of isolated, distinct islands. Instead, local optima are often connected by continuous, winding "valleys" of low loss. However, because these valleys are highly non-linear, a direct linear interpolation between two points on the valley floor cuts straight through the surrounding high-loss ridges.
Keller Jordan et al. formalize this drop in performance as the interpolation barrier in REPAIR: REnormalizing Permuted Activations for Interpolation Repair. When independently trained models are averaged, the internal activations of the intermediate network suffer from severe variance starvation, causing the model's representational capacity to collapse.
Symmetries and Coordinate Scrambling
Why do two networks solving the same problem end up in completely different regions of the parameter space? The primary culprit is permutation invariance.
In any hidden layer of a neural network, you can swap the positions of two neurons without changing the network's overall function, provided you also swap their corresponding incoming and outgoing weights. For a network with thousands of neurons, there are an astronomical number of functionally identical ways to arrange the parameters.
When two networks are trained from different random initializations, they converge to different "basins" or "wedges" of the loss landscape, as described by Stanislaw Jastrzebski in Large Scale Structure of Neural Network Loss Landscapes. Because their internal neurons are ordered differently, their weight spaces are functionally unaligned. Naively averaging their parameters is akin to averaging two sentences written in different languages; even if they convey the same meaning, a word-by-word average results in gibberish. This mismatch leads to what Samuel Horváth et al. term the "vanishing feature" phenomenon in Vanishing Feature: Diagnosing Model Merging and Beyond, where the unique features learned by each parent model cancel each other out during naive interpolation.
Dissolving the Barrier via Alignment
If the loss barrier is merely an artifact of coordinate misalignment, can we eliminate it by "unscrambling" the permutations?
Rahim Entezari et al. advanced this exact conjecture in The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, suggesting that if neurons are properly aligned, independently trained models might actually reside in the same low-loss basin. This insight led to the development of "re-basining" algorithms. For instance, Samuel Ainsworth et al. introduced Git Re-Basin: Merging Models modulo Permutation Symmetries, demonstrating that permuting the weights of one model to match the internal ordering of another can frequently eliminate the loss barrier entirely, allowing zero-loss model merging.
Independently Trained Models:
Model A (Basin 1) ====== Naive Interpolation (High Loss Barrier) ======> Model B (Basin 2)
With Re-Basining:
Model A (Basin 1) <== Permutation Alignment == Model B' (Teleported to Basin 1)
====== Linear Path (Low Loss) ======>
Recent work has extended this concept beyond discrete permutations. In Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion, Binchi Zhang et al. identify continuous rotation symmetries within Transformer self-attention layers, proving that continuous geometric alignment is just as vital as discrete neuron matching for eliminating the fusion barrier.
From a theoretical perspective, the height of this barrier is fundamentally tied to network width. Keyao Zhan et al. demonstrate in Analyzing the Role of Permutation Invariance in Linear Mode Connectivity that as network width $m$ increases, the post-alignment loss barrier undergoes a predictable decay at a rate of $O(m^{-1/2})$, bypassing the curse of dimensionality.
Shared Initializations and Model Soups
The loss barrier behaves completely differently when models share a common starting point. If multiple models are fine-tuned from the same pre-trained checkpoint, they typically remain within the same optimization basin. Because they share a coordinate system from the start, the linear path between them remains entirely within a low-loss valley.
This property underpins the success of Model Soups, a technique introduced by Mitchell Wortsman et al. in Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. By averaging the weights of various models fine-tuned with different hyperparameters from a shared initialization, practitioners can construct a single model that generalizes better than any individual member, without incurring any inference overhead.
However, this connectivity is fragile. As noted by Linara Adilova et al. in Landscaping Linear Mode Connectivity, if the learning rate during fine-tuning is too high, or if the models are optimized for diametrically opposed objectives, the trajectories can escape the shared basin, causing the linear mode connectivity to break down and the barrier to re-emerge.
Applications: Merging, Shuffling, and Federated Learning
Understanding and managing the linear mode connectivity barrier has unlocked several practical capabilities across deep learning:
- Advanced Model Merging: Algorithms like PLeaS, developed by Anshul Nasery et al. in PLeaS - Merging Models with Permutations and Least Squares, combine permutation alignment with layer-wise least-squares optimization to merge highly specialized models without losing representational power. Similarly, the cycle-consistent framework proposed by Daniele Baieri et al. in $C^2M^3$: Cycle-Consistent Multi-Model Merging prevents error accumulation when aligning multiple models simultaneously.
- Forced Basin Coherence: Instead of fixing alignment post-training, some methods force models to stay in the same basin during training. The WASH framework by Louis Fournier et al. (WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average) randomly shuffles a small fraction of weights between ensemble members during optimization, keeping them within a single averageable basin.
- Federated Learning: In decentralized training, local models often drift into isolated regions of the loss landscape. Zexi Li et al. address this in Improving Group Connectivity for Generalization of Federated Deep Learning by using historical global models as anchors, leveraging geometric transitivity to ensure that local updates remain linearly connectable and can be safely averaged by the central server.
Key Foundational Work
Large Scale Structure of Neural Network Loss Landscapes (2019) — contributed 3 concept elements to this direction.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018) — contributed 1 concept elements to this direction.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018) — contributed 1 concept elements to this direction.