The Connected Geometry of Neural Network Loss Landscapes

This direction is split into other directions: loss landscape connectivity. Its last version stays readable and citable here.

Version 1 of 174 papers101 conceptsfrozen 2026-10-04

In classical optimization, we often visualize the loss landscape of a non-convex function as a rugged mountain range. In this mental model, different training runs—starting from different initializations—land in isolated "basins" of low loss, separated by high-energy ridges. To move from one solution to another, you would seemingly have to climb over a steep mountain of high error.

However, research into mode connectivity has revealed that this intuition is largely incorrect for deep neural networks. Instead of isolated islands, local minima are part of a sprawling, interconnected manifold of high performance.

The Discovery of Low-Loss Valleys

The foundational insight, popularized by Timur Garipov and Felix Draxler, is that while a straight line between two independent solutions usually passes through a region of high loss, a slightly curved path does not. In Essentially No Barriers in Neural Network Energy Landscape, Draxler et al. demonstrated that one can find continuous paths between disparate minima where the loss remains nearly constant.

This suggests that the "modes" (local optima) found by Stochastic Gradient Descent (SGD) are not discrete points but are connected by high-accuracy "valleys." Garipov et al. showed that even a simple polygonal chain with a single bend is often enough to link two optima without any significant drop in performance. More recently, Jie Ren and colleagues have extended this to Bézier surfaces, showing that models can be connected via 2D manifolds, providing a much richer map of the landscape than a simple 1D path.

Why the Landscape is Connected

Why does this connectivity exist? The answer lies in the massive over-parameterization and inherent symmetries of neural networks.

  1. Functional Equivalence: Many different weight configurations produce the exact same mathematical function. Matthew Farrugia-Roberts proved in Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks that in networks with redundant neurons, all parameters implementing the same function are connected by piecewise-linear paths.
  2. Symmetry and Redundancy: Neural networks are invariant to permutations of neurons in a hidden layer. This redundancy creates "valleys" of equivalent solutions. Daniel Lengyel developed the GENNI tool to visualize these symmetries, showing that functionally equivalent parameter sets form topologically connected regions.
  3. Structural Properties: Quynh Nguyen 0001 provided theoretical evidence in On Connected Sublevel Sets in Deep Learning that for wide enough networks, the loss surface has no "bad" local valleys; every point has a non-increasing path to a global minimum.

Entropic Barriers: The Invisible Walls

If these low-loss paths exist, why doesn't SGD naturally wander between them? Luca Di Carlo identifies a phenomenon called entropic confinement. In Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks, Di Carlo explains that while the loss might be flat along a path, the curvature of the landscape often increases at the midpoint between two solutions.

This creates an "entropic barrier." Even if the energy (loss) is low, the region is "narrower" or more rugged, causing the noise in SGD to push the model back toward the wider, flatter endpoints. This explains why models appear trapped in basins despite the existence of connecting tunnels.

From Theory to Practice: Model Fusion and Ensembling

Understanding mode connectivity has transformed how we combine and train models.

  • Fast Geometric Ensembling (FGE): Instead of training multiple models from scratch to create an ensemble, Garipov et al. use the geometry of the landscape to "collect" high-performing models along a low-loss path, creating an ensemble in the time it takes to train a single model.
  • Deep Model Fusion: As surveyed by Weishi Li and Han Hu, researchers now use mode connectivity to merge independently trained models. By finding the low-loss path between two models, we can interpolate their weights to find a single model that inherits the strengths of both.
  • Star Domains: Recent work on Star Domain Topology suggests that the landscape might even contain "universal hubs"—central models that maintain low-loss linear connectivity to a wide variety of other solutions.

Universal Connectivity

Crucially, this connectivity is not limited to simple architectures. Yongding Tian and colleagues have shown in Connecting Independently Trained Modes via Layer-Wise Connectivity that these paths exist even in modern Transformers and MobileNets. Furthermore, Jakub Vrábel and Yaron Oz have discovered that mode connectivity even exists in the input space, where continuous low-loss paths connect different images that share the same classification.

By shifting our view from isolated points to a globally connected web, mode connectivity provides a powerful framework for understanding why deep learning is so robust and how we can better navigate its complex landscapes.

Papers in this version

Showing 60 of 74 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_connected_geometry_of_neural_network_7e048a7d,
  title = {The Connected Geometry of Neural Network Loss Landscapes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-connected-geometry-of-neural-network-loss-landscapes/rdr_7e048a7d3e8a429685c8cf2baecafb00},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_connected_geometry_of_neural_network_7e048a7d_v1,
  title = {The Connected Geometry of Neural Network Loss Landscapes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-connected-geometry-of-neural-network-loss-landscapes/rdr_7e048a7d3e8a429685c8cf2baecafb00/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}