Continuous Low Loss Manifolds Connecting Neural Network Optima

24 concepts14 papers

When we visualize the loss landscape of a deep neural network, our mental model is often a rugged mountain range. We imagine distinct optimization runs starting from different random initializations, navigating down treacherous slopes, and finally settling into isolated, bowl-shaped valleys. In this classical view, moving from one local minimum to another requires climbing over high-loss ridges.

However, a growing body of research reveals that this intuition is fundamentally flawed. Instead of isolated basins, the high-dimensional parameter space of neural networks contains a globally connected web of low-loss regions. This phenomenon, known as mode connectivity, suggests that independently trained models are merely different points on a shared, continuous manifold of high performance.

The Discovery of Low-Loss Paths

The shift in perspective began with the empirical discovery that separate minima are not separated by energy barriers. Draxler et al. demonstrated that if you take two independently trained neural network solutions, you can find a continuous, essentially flat path connecting them along which the loss remains consistently low.

Why do these paths exist? Early explanations assumed infinite network width, but subsequent theoretical frameworks brought this phenomenon down to realistic dimensions. Kuditipudi et al. provided a rigorous mathematical foundation for mode connectivity by defining properties like "dropout stability" and "noise stability," proving that simple, low-loss paths can be constructed using just a few linear segments in multilayer networks. Further refining the architectural requirements, Nguyen et al. proved that interconnectivity does not require massive over-parameterization across the entire network; it is guaranteed as long as the last two hidden layers have a width proportional to the square root of the number of training samples ($O(\sqrt{N})$).

The Geometry of Connectivity: Wedges and Volumes

While initial work focused on finding one-dimensional strings between pairs of models, the true geometry of the loss landscape is far more expansive. Rather than narrow tunnels, the low-loss regions form multi-dimensional manifolds.

To build intuition for how multiple independent solutions can be connected simultaneously, Jastrzebski proposed a phenomenological model where the loss landscape is treated as a union of high-dimensional "wedges." These wedges radiate from a central initialization point and, due to the sheer dimensionality of the parameter space, intersect with one another. This intersection creates a connected web of low-loss subspaces where entire sets of independently trained solutions can be linked.

Classical View:                      Manifold View:
   Loss                                 Loss
    ▲                                    ▲
    │   /\        /\                     │   /\            /\
    │  /  \      /  \                    │  /  \__________/  \
    │ /    \____/    \                   │ /    \        /    \
    └───────────────────►                └─────────────────────►
     Min A      Min B                     Min A  (Low-Loss) Min B
   (Isolated Basins)                             Path

Expanding on this, Benton et al. showed that high-performance regions extend across many dimensions—often ten or more in standard convolutional architectures—forming expansive multi-dimensional volumes or simplexes. Similarly, Ren et al. utilized nonlinear Bézier surfaces to demonstrate that multiple models can be mapped onto a continuous 2D sheet of high-performing parameters. This high-dimensional nature is also evident when analyzing the convex hull of multiple training trajectories; as Yunis et al. observed, these hulls span a substantial subspace close in dimension to the number of endpoints, rather than collapsing into a degenerate, low-dimensional space.

The Paradox of Basin Confinement

If the loss landscape is a single, globally connected manifold of low loss, a natural question arises: Why does standard optimization like Stochastic Gradient Descent (SGD) remain confined to a single basin? Why don't our models naturally drift along these flat paths during training?

This "basin confinement paradox" was resolved by Di Carlo et al. through the concept of entropic confinement. The authors demonstrated that while the connecting paths are energetically flat (maintaining low loss), they are entropically rugged. As you move away from the endpoints along the path, the local curvature of the landscape rises sharply. This creates invisible, entropic barriers: the noisy dynamics of SGD are naturally repelled by high-curvature regions and pushed back toward the wider, lower-curvature basins of the original endpoints.

This structural understanding is complemented by insights from Morse theory. Akhtiamov et al. discovered that standard optimizers do not actually come to rest in isolated local minima. Instead, they land in high-index saddle points. These saddle points naturally possess many directions of negative or zero curvature, providing the geometric exit routes that make mode connectivity possible in the first place.

Mapping and Exploiting the Manifold

Understanding this global connectivity allows researchers to better map and utilize the parameter space. To probe these structures directly, Jules et al. used thermal noise (Langevin dynamics) as a geometric probe, revealing that the low-loss manifold is bounded by exponential "walls" rather than quadratic wells, with its dimensionality dictated by the number of data points near the classification boundary.

The nature of these paths can also vary by architecture and task:

  • Graph Neural Networks (GNNs): Han et al. found that while standard CNNs often exhibit linear mode connectivity, GNN minima are typically connected by non-linear paths (like quadratic Bézier curves), driven heavily by the underlying graph structure.
  • Pre-trained Language Models (PLMs): Qian et al. showed that fine-tuning naturally leads to well-connected solutions, and pre-training acts as a crucial "glue" that pulls different task solutions closer together in parameter space.

Finally, this structural insight enables practical applications. Algorithms like the Low-Loss Path Finding (LLPF) routine developed by Tian et al. can reliably discover these tunnels across diverse architectures like Transformers by focusing on layer-wise connectivity. Moreover, by sampling diverse models from the expansive low-loss volumes discovered via mode connectivity, frameworks like those proposed by Benton et al. can construct highly efficient ensembles that vastly outperform traditional deep ensembles in accuracy and robustness.

As Wilson notes, while many deep learning phenomena can be explained by classical generalization frameworks, mode connectivity remains a distinct, foundational property of neural network loss landscapes that opens up entirely new ways to understand, merge, and manipulate deep models.

Key Foundational Work

Large Scale Structure of Neural Network Loss Landscapes (2019) — contributed 3 concept elements to this direction.
Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets (2019) — contributed 2 concept elements to this direction.
Essentially No Barriers in Neural Network Energy Landscape (2018) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_loss_landscape_connectivity_58cd4107,
  title = {loss landscape connectivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/loss-landscape-connectivity/rdr_58cd410739ee40be9286c3ee07f17b67},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_loss_landscape_connectivity_58cd4107_v1,
  title = {loss landscape connectivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/loss-landscape-connectivity/rdr_58cd410739ee40be9286c3ee07f17b67/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake