Linear Mode Connectivity in Deep Learning Loss Landscapes

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 246 papers47 conceptsfrozen 2026-10-04

Neural network optimization is often visualized as a treacherous journey through a rugged mountain range filled with millions of local minima and "traps." However, a growing body of research suggests this intuition is largely incorrect. Instead of isolated peaks, the loss landscapes of modern deep networks appear to be composed of vast, interconnected "valleys" or basins of low loss.

The study of Linear Mode Connectivity (LMC) investigates this phenomenon: the surprising fact that two different neural networks, even those starting from different initializations or trained on different data, can often be connected by a simple straight line in parameter space without ever encountering a region of high loss.

The Geometry of the Straight Path

The primary tool for probing these landscapes is linear interpolation. If we have two sets of trained weights, $w_A$ and $w_B$, we can evaluate the model's performance at any point along the line connecting them: $w(\alpha) = (1-\alpha)w_A + \alpha w_B$.

Early work by Ian Goodfellow and Oriol Vinyals in Qualitatively characterizing neural network optimization problems revealed that for many successful models, there is a smooth, nearly convex path from a random initialization to a solution. This suggests that the "difficulty" of training isn't about escaping local minima, but rather navigating the scale and conditioning of these broad basins.

As Tomaso Poggio and Qianli Liao argue in Theory II: Landscape of the Empirical Risk in Deep Learning, overparameterization creates a landscape dominated by "degenerate" global minima. Because the network has far more parameters than necessary, there isn't just one way to solve a task; there is a whole manifold of solutions.

Why Symmetry Matters

If you train two networks from different random seeds, a direct linear interpolation usually fails—you hit a "loss barrier" where the error spikes. For a long time, this was taken as evidence that the models were in different, isolated basins.

However, Bo Zhao and colleagues show in Understanding Mode Connectivity via Parameter Space Symmetry that this barrier is often an illusion caused by permutation symmetry. Because the neurons in a hidden layer can be reordered without changing the network's output, $w_A$ and $w_B$ might represent the exact same functional solution but with their "internal wiring" shuffled.

When you account for this—by permuting the neurons of one model to align with the other—the loss barrier often vanishes. Frederik Benzing and Johannes von Oswald demonstrated in Random initialisations performing above chance and how to find them that even two random initializations can be shown to live in the same valley if you find the right permutation.

Pre-training as a Shared Foundation

One of the most robust ways to achieve linear mode connectivity is through pre-training. When two models are fine-tuned from the same pre-trained checkpoint, they almost always remain linearly connectable.

In Exploring Mode Connectivity for Pre-trained Language Models, Cheng Qian and Maosong Sun show that pre-training acts as a "glue" that pulls different task solutions into a shared region of the parameter space. This shared foundation is so powerful that it even enables "model merging"—averaging the weights of two models fine-tuned on different tasks to create a single model that can perform both.

This connectivity has practical implications for privacy and lifelong learning. Arun Ganesh and Sewoong Oh found that public pre-training is essential for private training because it places the model in a "good basin" where optimization is stable enough to withstand the noise required for differential privacy (Why Is Public Pretraining Necessary for Private Model Training?).

Flatness and Generalization

The "shape" of these basins—specifically their flatness—is a key predictor of how well a model will generalize to unseen data.

Nitish Shirish Keskar and Dheevatsa Mudigere famously showed in On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima that large-batch training tends to converge to "sharp" minima (narrow valleys), which generalize poorly. In contrast, small-batch SGD finds "flat" minima (wide valleys) that are more robust to shifts in the data distribution.

This relationship between landscape geometry and generalization is further explored by Sanket Vaibhav Mehta in An Empirical Investigation of the Role of Pre-training in Lifelong Learning, who found that pre-trained models naturally settle into flatter regions, making them more resistant to "catastrophic forgetting" when learning new tasks.

When Connectivity Breaks

Connectivity is not universal. It can be broken by:

  1. Optimization Anomalies: The "Slingshot Effect" described by Vimal Thilak and Omid Saremi involves sudden late-stage loss spikes that can kick a model into a completely different basin, often triggering "grokking" (The Slingshot Effect).
  2. Structural Bottlenecks: Linara Adilova and Maksym Andriushchenko discovered that while individual layers are almost always connectable, the interaction between layers—particularly in the middle of the network—is what creates the overall loss barrier (Layer-wise linear mode connectivity).

By visualizing and measuring these paths, researchers are moving away from treating neural networks as "black boxes" and toward a geometric understanding of how they learn, why they generalize, and how they can be combined.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 41 papers
  2. Version 1 · 2026-10-04 · 46 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_weight_interpolation_linear_mode__6657f21d,
  title = {linear weight interpolation (linear mode connectivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-weight-interpolation-linear-mode-connectivity/rdr_6657f21dac04438b98519b6e0f03abc2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_mode_connectivity_in_deep_learnin_6657f21d_v1,
  title = {Linear Mode Connectivity in Deep Learning Loss Landscapes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-mode-connectivity-in-deep-learning-loss-landscapes/rdr_6657f21dac04438b98519b6e0f03abc2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}