Dynamic Preconditioning for Evolving Optimization Landscapes

16 concepts7 papers1 proposal

Preconditioning is the art of changing coordinates to make optimization easier. Imagine trying to find the lowest point of a steep, narrow valley. Standard gradient descent methods will bounce back and forth between the walls rather than moving along the floor. A preconditioner warps the space, transforming this elongated, poorly conditioned valley into a perfectly round, easily traversable bowl.

If you know the shape of the landscape beforehand, you can construct a static preconditioning matrix to perform this coordinate transformation once and for all. But what happens when the landscape changes as you walk through it?

In many complex, real-world problems, the underlying geometry is not static. As an optimization algorithm iterates, the local landscape shifts dynamically. This realization has driven a research direction focused on adaptive and variable preconditioning—updating the preconditioner on-the-fly to match the evolving state of the system.

The Failure of Static Preconditioning

Static preconditioners fail when a problem's mathematical properties change over time or iterations. This mismatch occurs across several distinct domains:

The obvious solution is to update the preconditioner at every step. However, doing so naively introduces a fundamental mathematical conflict.

Why Changing Preconditioners Breaks Standard Solvers

Classical iterative solvers—such as the standard Conjugate Gradient (CG) method—rely on a strict mathematical assumption: the preconditioner must remain fixed throughout the execution of the solver.

To understand why, consider how CG works. It constructs a sequence of search directions that are mutually orthogonal with respect to the system matrix and the preconditioner. This orthogonality creates a "memory" that prevents the algorithm from re-searching directions it has already covered, guaranteeing fast convergence.

If you alter the preconditioner mid-stream, you change the underlying metric of the space. The previously computed search directions lose their mutual orthogonality. The algorithm's memory is effectively wiped, causing it to lose track of its history. This breakdown can lead to severe convergence issues, stalling, or outright divergence.

The Solution: Flexible Solvers and Continuous Flows

To safely adapt a preconditioner without destroying algorithmic stability, researchers have developed two primary methodologies: flexible Krylov subspace solvers and continuous dynamic flows.

1. Flexible Krylov Solvers

Instead of forcing a changing preconditioner into a rigid solver, one can use flexible variants of Krylov subspace methods. As highlighted by Tara N. Sainath et al. in Accelerating Hessian-free optimization for Deep Neural Networks by implicit preconditioning and sampling, flexible solvers mathematically tolerate a preconditioner that changes from one inner iteration to the next (such as an approximate Hessian updated via L-BFGS statistics). They achieve this by storing extra vectors to explicitly correct for the loss of orthogonality, preserving convergence at the cost of a slight memory overhead.

2. Preconditioners as Continuous Dynamical Systems

A more recent, theoretical framework treats the preconditioner not as a sequence of discrete, abrupt updates, but as a continuous trajectory.

In Transformed Primal-Dual Methods with Variable-Preconditioners, Jingrong Wei et al. model the evolution of the preconditioner as a "dynamic flow" governed by differential equations. By treating the preconditioner updates as a continuous dynamical system, the algorithm smoothly warps the optimization space to track the changing geometry of nonlinear problems. This continuous formulation bridges the gap between discrete computer steps and continuous physical models, ensuring that the mathematical shortcuts taken by the preconditioner do not induce numerical instability.

Discrete Updates:   [Preconditioner A]  ==> abrupt jump ==>  [Preconditioner B]  (Breaks Orthogonality)

Continuous Flow:    [Preconditioner A]  ~~~~ smooth flow ~~~~> [Preconditioner B]  (Maintains Stability)

The Payoff of Adaptivity

When the solver is designed to handle a changing metric, adaptive preconditioning unlocks dramatic performance gains:

The paradigm of adaptive preconditioning continues to show promise in expanding domains, from path-following algorithms in convex quadratic programming (Renato D. C. Monteiro et al.) to time-implicit reaction-diffusion equations (Shuming Liu et al.). By abandoning the assumption of a fixed preconditioner and embracing dynamic metrics, modern numerical methods can remain tightly coupled to the fluid, shifting geometries of complex systems.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adaptive_preconditioning_fca95ed5,
  title = {adaptive preconditioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-preconditioning/rdr_fca95ed5f82546dfa488892ee6c590ea},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_adaptive_preconditioning_fca95ed5_v1,
  title = {adaptive preconditioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-preconditioning/rdr_fca95ed5f82546dfa488892ee6c590ea/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake