Linear Mode Connectivity in Deep Learning

43 concepts41 papers1 proposal

When we train a deep neural network, optimization algorithms navigate a high-dimensional loss landscape containing millions or billions of parameters. A classical, intuitive view of non-convex optimization suggests that this landscape is dotted with isolated, distinct valleys—local minima separated by high-loss energy barriers. Under this view, running stochastic gradient descent (SGD) twice with different random seeds or data orderings should land the model in completely separate valleys.

However, a profound structural property has emerged from empirical study: many of these seemingly isolated minima are actually connected by continuous paths of low loss. This phenomenon is known as Linear Mode Connectivity (LMC). Understanding why and when these paths exist provides deep insights into how neural networks generalize, how different architectures behave, and how internal representations align.

Measuring the Landscape via Interpolation

To understand if two distinct model solutions reside in the same low-loss region, researchers use weight space interpolation. Given two weight configurations, $W_1$ and $W_2$ (which might represent a model at different stages of training, or two models trained on different subsets of data), we can evaluate the loss along a straight-line path:

$$W(\alpha) = (1 - \alpha)W_1 + \alpha W_2 \quad \text{for} \quad \alpha \in [0, 1]$$

If the loss along this path remains consistently low without encountering a "loss barrier," the two endpoints are said to be linearly mode connected.

Early exploration of this geometry by Keskar et al. utilized linear interpolation to visualize the differences between optimization regimes. They demonstrated that large-batch training tends to land in sharp minima, while small-batch training finds flatter valleys that generalize better. Building on this, Sagun et al. discovered that large-batch and small-batch methods do not necessarily settle into entirely disconnected valleys. Instead, they often converge to different points within the same wide, flat basin, meaning they are connected by a path with no loss barrier.

To formalize this analysis, researchers like Juneja et al. utilize metrics like the convexity gap, which explicitly measures the maximum loss barrier encountered along the interpolation vector rather than just checking a single midpoint.

Loss
 ^
 |       Isolated Minima (Barrier)            Linear Mode Connectivity
 |             / \                                   
 |            /   \                                  
 |  _________/     \_________             _______________________
 | /                         \           /                       \
 |/                           \         /                         \
 +----------------------------->       +----------------------------->
W1                            W2      W1                            W2
                Weight Space                          Weight Space

The Permutation Obstacle and Hidden Symmetries

If you train two neural networks completely independently from different random initializations, a naive linear interpolation between them almost always reveals a massive loss barrier. Does this mean independent SGD solutions occupy isolated islands?

Not necessarily. Neural networks possess vast permutation invariances. For example, if you swap the positions of two hidden neurons in a layer and adjust the corresponding incoming and outgoing weights, the mathematical function represented by the network remains completely identical. However, in the raw parameter space, this swap moves the model to a completely different coordinate.

Imagine Model A learns a feature in Neuron 1, while Model B learns the exact same feature in Neuron 2. If you linearly interpolate their weights directly, the midpoint model will dilute both neurons, destroying the feature representation and causing a massive spike in loss.

To resolve this, Entezari et al. proposed that the energy barriers observed during interpolation are largely an artifact of ignoring these symmetries. By finding a permutation that aligns the neurons of Model A with Model B before interpolating, the loss barrier frequently vanishes. Zhao et al. provided a theoretical foundation for this, bounding the loss barrier mathematically by the curvature of the parameter space and showing how symmetry-induced paths can bridge seemingly disparate solutions.

When a perfectly straight line cannot bridge two models even after alignment, non-linear paths often can. The authors of Pittorino et al. demonstrated that by factoring out symmetries, high-quality flat minima form a continuous, connected "octopus-shaped" manifold. Researchers can map these complex geometries using parameterized curves, such as quadratic Bézier curves, to find smooth paths through rugged terrain, as explored by Schürholt et al. and Cheng et al..

What Connectivity Reveals About Model Logic

Linear mode connectivity is more than a geometric curiosity; it serves as a structural diagnostic tool to understand the internal "logic" of deep learning models.

Generalization Strategies

Does achieving identical accuracy on a test set mean two models have learned the same underlying rules? Juneja et al. used weight-space connectivity to show that text classifiers often fall into distinct, disconnected basins. Crucially, these geometric basins directly correspond to different linguistic generalization strategies—such as relying on simple word-overlap heuristics versus parsing complex syntax. If two models are linearly mode connected, they tend to share the same internal logic; if they are disconnected, they solve the problem differently.

The Role of Pre-training

Pre-trained language and vision models have revolutionized machine learning, and LMC helps explain why. Qian et al. demonstrated that pre-training acts as a geometric "glue." When models are initialized from a shared pre-trained checkpoint, the subsequent fine-tuned solutions for a task remain tightly bound within the same low-loss basin. Furthermore, Mehta et al. showed that pre-training fundamentally shifts the geometry toward flatter minima, making models significantly more resilient to catastrophic forgetting during continual learning.

Mechanistic Similarity in Unlearning

When removing sensitive data from a trained model, how do we know if an unlearning algorithm truly erased the information or merely masked it? By applying weight-space interpolation, Cheng et al. revealed that various machine unlearning formulations, despite having different mathematical setups, guide models into the exact same flat region of the loss landscape, proving their mechanistic similarity.

Diagnosing Optimization Anomalies

LMC also provides a window into the failures and hidden dynamics of optimization algorithms.

  • The Stability Gap: In incremental learning, models often suffer a temporary drop in performance on old tasks when introduced to new data. Soutif utilized LMC to show that a direct, low-loss path to the joint solution exists in the landscape, but standard SGD optimization takes a high-loss "detour" away from it, creating the stability gap.
  • Reinforcement Learning Fragility: Deep reinforcement learning agents are notoriously sensitive to hyperparameter choices. By interpolating policy parameters, Rahn et al. mapped out the "return landscape" of continuous control tasks, distinguishing between robust policies that sit in wide, well-connected basins and fragile policies isolated in noisy neighborhoods.
  • Pruning at Initialization: Finding sparse, high-performing subnetworks before training begins is notoriously difficult due to chaotic loss surfaces. McDermott et al. demonstrated that dataset distillation can be used to identify sparse subnetworks that maintain strong linear mode connectivity under different training noise regimes, ensuring they remain stable from day one.

Key Foundational Work

Qualitatively characterizing neural network optimization problems (2015) — contributed 1 concept elements to this direction.
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2017) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 41 papers
  2. Version 1 · 2026-10-04 · 46 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_weight_interpolation_linear_mode__6657f21d,
  title = {linear weight interpolation (linear mode connectivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-weight-interpolation-linear-mode-connectivity/rdr_6657f21dac04438b98519b6e0f03abc2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_linear_weight_interpolation_linear_mode__6657f21d_v2,
  title = {linear weight interpolation (linear mode connectivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-weight-interpolation-linear-mode-connectivity/rdr_6657f21dac04438b98519b6e0f03abc2/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake