Escaping Local Minima Through Dimension Expansion

26 concepts14 papers1 proposal

Imagine a ball resting at the bottom of a deep crater on a two-dimensional sheet. To escape this local minimum and find a lower valley elsewhere, the ball must somehow acquire enough energy to roll up and over the high surrounding rim.

But what if we could pull the sheet into a third dimension? By warping the sheet vertically, that insurmountable rim can be deformed into a mountain pass—a saddle point where the landscape curves downward in the new direction. The ball no longer needs to climb uphill; it can simply roll sideways through the pass to escape.

This is the core intuition behind dimension lifting (or dimension expansion). Across machine learning, spatial localization, geometry processing, and physics simulations, optimization problems frequently get trapped in suboptimal local minima. By intentionally embedding these problems into higher-dimensional spaces, researchers are discovering that we can fundamentally alter the topology of the optimization landscape, turning dead ends into open pathways.


The Mathematics of Spurious-to-Saddle Conversion

Why does adding dimensions dissolve local minima? The answer lies in the curvature of the loss landscape, which is mathematically captured by the Hessian matrix.

For a point to be a stable local minimum, the landscape must curve upward in every possible direction of movement. In other words, all eigenvalues of the Hessian matrix must be positive. When we over-parametrize a problem or lift its variables into a higher-dimensional space, we introduce new degrees of freedom. This appends new rows and columns to the Hessian matrix.

Crucially, if the lifting process introduces even one direction of negative curvature at a previously trapping point, that local minimum is instantly destabilized. It transforms into a strict saddle point—a critical point where the gradient is zero, but at least one direction leads downward. Standard gradient-based optimizers can easily exploit this negative curvature to break away.

Low-Dimensional Space                  Higher-Dimensional Space
     (Local Minimum)                       (Strict Saddle Point)

         |   |                                  \     /
         |   |                                   \   /  <- Escape Route
        /     \                                   \ /
       (  Ball )                                 (Ball)

Ziye Ma et al. formally proved this mechanism in Over-parametrization via Lifting for Low-rank Matrix Sensing: Conversion of Spurious Solutions to Strict Saddle Points. They demonstrated that while over-parametrization leaves spurious solutions as stationary points, the lifting process modifies the Hessian to guarantee negative eigenvalues, providing a reliable escape route to the global optimum.

A similar landscape transformation occurs in biologically inspired neural networks. El Mehdi Achour et al. in Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? and Francesco Innocenti et al. in Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? showed that predictive coding networks fundamentally reshape the loss landscape. They convert flat, degenerate non-strict saddles—where standard backpropagation stalls due to vanishing gradients—into highly unstable strict saddles. Furthermore, Francesco Innocenti et al. demonstrated in Understanding Predictive Coding as a Second-Order Trust-Region Method that this structural modification leverages curvature information to make flat plateaus unstable, accelerating the optimization process.


Untangling Physical and Geometric Folds

In physical simulations and spatial geometry, local minima often correspond to physically impossible "folded" or "tangled" states.

Consider wireless sensor network localization, where the goal is to determine the 2D coordinates of sensors based on distance measurements. Optimization algorithms frequently get trapped in configurations where the estimated network is folded over itself. Jieqi Yu et al. tackled this in Dimension expansion and customized spring potentials for sensor localization by treating the 2D layout as a 3D simulation. The extra spatial dimension allows the network to "flip" through the third dimension to untangle itself, converting insurmountable 2D traps into unstable 3D saddle points.

A parallel breakthrough was achieved by Juri Sidorenko et al. in Self-Calibration for the Time-of-Arrival Positioning. By introducing temporary auxiliary coordinates to expand the degrees of freedom, they reduced the failure rate of non-linear positioning solvers from over 60% to near-zero, allowing gradient solvers to bypass suboptimal geometric configurations.

This geometric untangling principle extends across several domains:


Surfing to Global Optima

Beyond escaping specific traps, dimension lifting can completely smooth out non-convex landscapes, enabling local search methods to act as global optimizers.

In computer vision, averaging 3D rotations to reconstruct scenes is notoriously non-convex and prone to local minima. Frank Dellaert et al. introduced Shonan Rotation Averaging in Shonan Rotation Averaging: Global Optimality by Surfing SO(p)n.. By lifting 3D rotations into higher-dimensional rotation manifolds (like 5D or 6D), they eliminated local minima entirely. This allows the algorithm to "surf" down a simplified landscape to the global optimum before projecting the solution back to 3D.

[3D Rotation Space]  ---> Lift to --->  [5D/6D Manifold]
 (Many Local Minima)                     (Smooth Landscape, No Minima)
                                                      |
                                           Surf to Global Optimum
                                                      |
[3D Global Solution] <--- Project <---   [Global Minimum]

Similarly, Ji Xu et al. in Benefits of over-parameterization with EM showed that treating known mixing weights as unknown variables in Gaussian Mixture Models expands the optimization manifold. This creates escape routes that allow the Expectation-Maximization algorithm to bypass spurious local optima.

This philosophy also reshapes how we solve inverse problems, extract complex structures, and train multi-modal systems:

Optimization is traditionally seen as a search within a fixed arena. Dimension lifting changes the arena itself. By stepping out of the target space, algorithms gain the geometric freedom required to render previously insurmountable barriers completely passable.

Key Foundational Work

Dimension expansion and customized spring potentials for sensor localization (2013) — contributed 3 concept elements to this direction.
Vessels as 4D Curves: Global Minimal 4D Paths to Extract 3D Tubular Surfaces (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spurious_to_saddle_conversion_c7acb387,
  title = {spurious-to-saddle conversion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spurious-to-saddle-conversion/rdr_c7acb38768d9443690326e683d0585cb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spurious_to_saddle_conversion_c7acb387_v1,
  title = {spurious-to-saddle conversion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spurious-to-saddle-conversion/rdr_c7acb38768d9443690326e683d0585cb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake