Initialization Sensitivity in Non-Convex Optimization

This direction is merged into another direction: initialization sensitivity (non-convex optimization sensitivity). Its last version stays readable and citable here.

Version 1 of 1207 papers216 conceptsfrozen 2026-10-04

In a perfectly convex world, optimization is simple: every path leads to the same global minimum. But modern machine learning rarely lives in such a world. Instead, we navigate "non-convex" landscapes—vast, rugged terrains filled with false peaks, narrow ridges, and countless local valleys.

In these environments, the most critical decision an algorithm makes isn't how it moves, but where it starts. This phenomenon, known as initialization sensitivity, means that two identical models, starting just a few millimeters apart on the loss surface, can converge to vastly different solutions—one a state-of-the-art predictor, the other a useless failure.

The Geometry of Local Optima

The fundamental challenge of non-convexity is that gradient-based solvers are "greedy." They follow the steepest path downward, which works well until they encounter a local minimum—a basin that is not the lowest point in the entire landscape but has no downward exit.

As Hailiang Liu notes in AEGD: Adaptive Gradient Descent with Energy, these landscapes are often so sensitive to initial conditions that even slight overshooting can cause a model to explode or take inefficient detours. This is particularly visible in structured problems like Gene Regulatory Network Inference, where Deborah Weighill and colleagues observe that without high-quality prior knowledge to guide the starting point, algorithms often settle into mathematically valid but biologically irrelevant solutions.

Learning Where to Start

If the starting point is so decisive, why do we often choose it at random? Recent research suggests we can do better by treating initialization as a learnable parameter itself.

In Learning to Initialize Gradient Descent Using Gradient Descent, Kartik Ahuja, Amit Dhurandhar, and Kush R. Varshney argue that we should move away from manual heuristics. By using a "meta-learning" approach, they show that we can optimize the initial state of a model to ensure faster convergence and higher-quality final solutions. This is a shift from optimizing the path to optimizing the origin.

Escaping Valleys with Stochasticity and Smoothing

When we cannot find a perfect starting point, we must find ways to prevent the algorithm from getting trapped in the first valley it encounters.

One powerful tool is stochasticity. Francesca Mignacco, Pierfrancesco Urbani, and Lenka Zdeborova demonstrate in Stochasticity helps to navigate rough landscapes that mini-batch noise acts as a "self-annealing" mechanism. This noise allows the algorithm to "jump" out of shallow local minima that would otherwise trap a deterministic solver.

Another approach is to "smooth" the landscape before the search begins. Luca Pasa proposes a pre-training strategy that uses a simpler model (like a Hidden Markov Model) to create a synthetic, simplified version of the data. By training on this "smoothed" landscape first, the model reaches a neighborhood of the parameter space that is much closer to the global optimum before it ever sees the "rugged" real-world data.

The Bi-Level Tuning Trap

Initialization sensitivity isn't just a problem for model weights; it also plagues the "meta-parameters" of our systems. When we tune hyperparameters—like the regularization strength in a Lasso model or the weights in a control policy—we are solving a "bi-level" optimization problem.

As Quentin Bertrand and Shane T. Barratt highlight in their respective works on implicit differentiation and auto-tuning, these outer loops are frequently non-convex. If you start your hyperparameter search with the wrong initial guess, your entire training pipeline may converge to a sub-optimal configuration, regardless of how much data you have.

Convex Relaxation: Changing the Map

Perhaps the most elegant way to handle initialization sensitivity is to change the landscape itself. Convex relaxation involves finding a "convex proxy" for a non-convex problem—essentially flattening the ridges and filling the valleys so that only one minimum remains.

In Phase Retrieval Meets Statistical Learning Theory, Sohail Bahmani and Justin Romberg achieve this by framing the signal recovery problem as finding an extreme point of a "complex polytope." By relaxing the constraints, they eliminate the need for a near-perfect starting point, allowing the algorithm to find the correct solution from almost anywhere in the space.

Summary of Intuition

Non-convex optimization is like trying to find the lowest point in a mountain range during a thick fog.

  1. The Problem: If you start in a small valley on the outskirts, you will never find the deep canyon in the center.
  2. The Solution: We can either learn to drop our "hikers" (algorithms) into better starting locations, use "noisy" movements to hop over small ridges, or mathematically "flatten" the mountains into a single bowl where every path leads to the bottom.

Papers in this version

Showing 60 of 207 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_initialization_sensitivity_in_non_convex_27a0b842,
  title = {Initialization Sensitivity in Non-Convex Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initialization-sensitivity-in-non-convex-optimization/rdr_27a0b842a52046acbb6241f7affe2b0b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_initialization_sensitivity_in_non_convex_27a0b842_v1,
  title = {Initialization Sensitivity in Non-Convex Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initialization-sensitivity-in-non-convex-optimization/rdr_27a0b842a52046acbb6241f7affe2b0b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}