The Linear Mode Connectivity Barrier in Neural Networks

93 concepts73 papers1 proposal

Suppose you train two identical neural network architectures on the exact same dataset, starting from different random seeds. Both models reach near-perfect accuracy. If you average their predictions—a standard ensembling technique—the resulting combination performs exceptionally well. But what happens if you average their parameters directly?

Intuitively, if both networks have found valid solutions to the same problem, their average parameter state might be expected to yield a reasonable intermediate solution. Instead, the resulting model typically performs no better than random chance.

In the optimization landscape of deep neural networks, this phenomenon is known as the linear mode connectivity (LMC) barrier or interpolation barrier. While high-performing solutions are often connected by complex, non-linear paths of low loss, a straight line drawn between them in parameter space almost always passes through a massive wall of high loss. Understanding why this barrier exists, when it disappears, and how to bypass it has become a central focus of research into the geometry of neural network loss landscapes.

The Geometry of the Barrier

To understand why naive weight averaging fails, we must look at the geometry of the loss surface. As shown by Timur Garipov et al. in Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, the loss landscape is not a collection of isolated, distinct islands. Instead, local optima are often connected by continuous, winding "valleys" of low loss. However, because these valleys are highly non-linear, a direct linear interpolation between two points on the valley floor cuts straight through the surrounding high-loss ridges.

Keller Jordan et al. formalize this drop in performance as the interpolation barrier in REPAIR: REnormalizing Permuted Activations for Interpolation Repair. When independently trained models are averaged, the internal activations of the intermediate network suffer from severe variance starvation, causing the model's representational capacity to collapse.

Symmetries and Coordinate Scrambling

Why do two networks solving the same problem end up in completely different regions of the parameter space? The primary culprit is permutation invariance.

In any hidden layer of a neural network, you can swap the positions of two neurons without changing the network's overall function, provided you also swap their corresponding incoming and outgoing weights. For a network with thousands of neurons, there are an astronomical number of functionally identical ways to arrange the parameters.

When two networks are trained from different random initializations, they converge to different "basins" or "wedges" of the loss landscape, as described by Stanislaw Jastrzebski in Large Scale Structure of Neural Network Loss Landscapes. Because their internal neurons are ordered differently, their weight spaces are functionally unaligned. Naively averaging their parameters is akin to averaging two sentences written in different languages; even if they convey the same meaning, a word-by-word average results in gibberish. This mismatch leads to what Samuel Horváth et al. term the "vanishing feature" phenomenon in Vanishing Feature: Diagnosing Model Merging and Beyond, where the unique features learned by each parent model cancel each other out during naive interpolation.

Dissolving the Barrier via Alignment

If the loss barrier is merely an artifact of coordinate misalignment, can we eliminate it by "unscrambling" the permutations?

Rahim Entezari et al. advanced this exact conjecture in The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks, suggesting that if neurons are properly aligned, independently trained models might actually reside in the same low-loss basin. This insight led to the development of "re-basining" algorithms. For instance, Samuel Ainsworth et al. introduced Git Re-Basin: Merging Models modulo Permutation Symmetries, demonstrating that permuting the weights of one model to match the internal ordering of another can frequently eliminate the loss barrier entirely, allowing zero-loss model merging.

Independently Trained Models:
Model A (Basin 1)  ====== Naive Interpolation (High Loss Barrier) ======> Model B (Basin 2)

With Re-Basining:
Model A (Basin 1)  <== Permutation Alignment ==  Model B' (Teleported to Basin 1)
                  ====== Linear Path (Low Loss) ======>

Recent work has extended this concept beyond discrete permutations. In Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion, Binchi Zhang et al. identify continuous rotation symmetries within Transformer self-attention layers, proving that continuous geometric alignment is just as vital as discrete neuron matching for eliminating the fusion barrier.

From a theoretical perspective, the height of this barrier is fundamentally tied to network width. Keyao Zhan et al. demonstrate in Analyzing the Role of Permutation Invariance in Linear Mode Connectivity that as network width $m$ increases, the post-alignment loss barrier undergoes a predictable decay at a rate of $O(m^{-1/2})$, bypassing the curse of dimensionality.

Shared Initializations and Model Soups

The loss barrier behaves completely differently when models share a common starting point. If multiple models are fine-tuned from the same pre-trained checkpoint, they typically remain within the same optimization basin. Because they share a coordinate system from the start, the linear path between them remains entirely within a low-loss valley.

This property underpins the success of Model Soups, a technique introduced by Mitchell Wortsman et al. in Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. By averaging the weights of various models fine-tuned with different hyperparameters from a shared initialization, practitioners can construct a single model that generalizes better than any individual member, without incurring any inference overhead.

However, this connectivity is fragile. As noted by Linara Adilova et al. in Landscaping Linear Mode Connectivity, if the learning rate during fine-tuning is too high, or if the models are optimized for diametrically opposed objectives, the trajectories can escape the shared basin, causing the linear mode connectivity to break down and the barrier to re-emerge.

Applications: Merging, Shuffling, and Federated Learning

Understanding and managing the linear mode connectivity barrier has unlocked several practical capabilities across deep learning:

Key Foundational Work

Large Scale Structure of Neural Network Loss Landscapes (2019) — contributed 3 concept elements to this direction.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018) — contributed 1 concept elements to this direction.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_mode_connectivity_barrier_044a4059,
  title = {linear mode connectivity barrier},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-mode-connectivity-barrier/rdr_044a4059b1ea46c3a44e68cbb6c32df1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_mode_connectivity_barrier_044a4059_v1,
  title = {linear mode connectivity barrier},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-mode-connectivity-barrier/rdr_044a4059b1ea46c3a44e68cbb6c32df1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake