Amortized Optimization via Lazy Curvature Updates

65 concepts40 papers1 proposal

When optimizing high-dimensional models, practitioners face a classic dilemma. First-order methods like Stochastic Gradient Descent (SGD) and Adam are computationally cheap per iteration, but they require thousands of steps because they are blind to the underlying geometry of the loss landscape. Second-order methods, such as Newton's method or Natural Gradient Descent, navigate complex landscapes efficiently by computing the Hessian or Fisher Information Matrix, but they are bottlenecked by the massive cost of computing and inverting these matrices at every step.

A rapidly growing body of research resolves this dilemma through a simple, elegant insight: local curvature changes much more slowly than the gradient. By decoupling the frequency of curvature updates from gradient updates—a strategy known as "lazy" or amortized second-order optimization—we can reap the fast convergence rates of second-order methods at a fraction of the computational cost.

The Principle of Curvature Stability

To understand why lazy updates work, consider the geometric difference between a gradient and a Hessian. The gradient represents the immediate slope of the loss surface at a exact point; it is highly sensitive and changes with every tiny adjustment to the model parameters. The Hessian, however, describes the broader curvature of the landscape—the orientation of valleys, ridges, and plateaus.

As Lesi Chen et al. demonstrate in Second-Order Min-Max Optimization with Lazy Hessians, this local curvature evolves slowly enough that a single Hessian snapshot remains a highly accurate guide over a sequence of multiple iterations. Instead of pausing at every step to recompute the full geometry, an optimizer can capture the Hessian at a specific snapshot point and reuse it to precondition a series of subsequent gradient updates.

Standard Second-Order:  [Hessian 1] -> [Step 1] -> [Hessian 2] -> [Step 2] -> [Hessian 3] -> [Step 3]
Lazy Second-Order:      [Hessian 1] -> [Step 1] -> [  Stale   ] -> [Step 2] -> [  Stale   ] -> [Step 3]

This temporal decoupling dramatically lowers the amortized cost of matrix factorization and inversion, which are typically the primary bottlenecks in second-order algorithms. El Mahdi Chayti et al. formalize this under a unified theoretical "helper framework" in Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods, proving that skipping Hessian updates does not sacrifice the superior convergence rates characteristic of exact second-order optimization.

Finding the Optimal Laziness Window

How long can an optimizer safely rely on a stale Hessian before its approximation error degrades the descent direction?

In Second-Order Optimization with Lazy Hessians, Nikita Doikov et al. show that the optimal balance between computational overhead and convergence speed is directly tied to the problem dimension $d$. By setting the Hessian update frequency to be proportional to the dimension (updating once every $d$ iterations), the authors achieve a provable square-root speedup in total arithmetic complexity compared to standard second-order methods.

For convex settings, Ilan Adler et al. introduce Proximal Shamanskii methods in New Proximal Newton-Type Methods for Convex Optimization, demonstrating that reusing the Hessian for a fixed interval preserves high-order convergence while drastically lowering factorization frequency. Similarly, in the context of nonlinear systems, Chengchang Liu et al. show in An Enhanced Levenberg-Marquardt Method via Gram Reduction that computing the full Gram matrix only at specific snapshot points minimizes the total theoretical complexity required to reach a target precision.

Eliminating Hardware Idle Time

While periodic updates reduce the number of matrix operations, traditional lazy methods still suffer from a practical flaw: the optimization process must periodically pause and block the hardware while a new Hessian is computed and inverted.

To eliminate this idle time, El Mahdi Chayti proposes a split-client framework in A Split-Client Approach to Second-Order Optimization. This approach completely decouples the fast gradient steps from the slow Hessian computations by allowing the curvature information to arrive asynchronously in the background. The optimization steps proceed continuously using the available stale curvature, achieving a wall-clock speedup proportional to the square root of the relative cost of the Hessian computation.

Timeline:
Main Thread:   [Step 1 (Stale H)] -> [Step 2 (Stale H)] -> [Step 3 (New H Arrives)]
Background:    [--- Computing New Hessian Matrix In Parallel ---]

This asynchronous pipeline mirrors distributed deep learning strategies. For instance, Jimmy Ba et al. demonstrate in Distributed Second-Order Optimization using Kronecker-Factored Approximations that because neural network curvature evolves slowly, using asynchronous, stale matrix inverses does not degrade optimization performance, allowing intensive curvature estimation to run concurrently with standard gradient updates.

Amortization in Deep Learning Preconditioners

The practical power of lazy curvature estimation is most visible in structural preconditioning methods designed for deep neural networks, such as K-FAC and Shampoo. These methods approximate the massive Hessian or Fisher Information Matrix using smaller Kronecker factors or tensor products, but still require expensive periodic matrix inversions or eigen-decompositions.

In Shampoo: Preconditioned Stochastic Tensor Optimization, Vineet Gupta et al. establish that performing matrix inversions or singular value decompositions periodically, rather than at every step, keeps the per-step wall-clock time competitive with standard first-order methods like Adam. As training progresses, the local geometry tends to stabilize even further.

In Convolutional Neural Network Training with Distributed K-FAC, J. Gregory Pauloski et al. exploit this stabilization by updating the preconditioner only every few hundred iterations, drastically reducing communication overhead in distributed environments. The authors show that moving redundant calculations into these infrequent decomposition stages allows the high computational cost of second-order optimization to be effectively hidden behind standard data-parallel operations.

The Staleness Trade-off

Despite its advantages, laziness introduces a fundamental trade-off: if the curvature information becomes too stale, the optimizer may choose suboptimal update directions.

As Lin Zhang et al. note in Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation, a significant lag between the state where the curvature was estimated and the current model parameters can lead to a mismatch in the preconditioned descent direction.

To mitigate this, Mohammad Mozaffari et al. introduce MKOR in MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates. They argue that frequently updating curvature information with cheap, approximate data (such as rank-1 updates) yields faster convergence than relying on perfectly calculated but highly stale curvature matrices.

Alternatively, single-author work by Dahun Shin, Sassha: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation, leverages the fact that in flat regions of the loss landscape, the Hessian remains remarkably consistent over multiple steps. This allows the optimizer to safely deploy lazy updates in flat regions while maintaining stability and generalization performance.

Ultimately, amortized second-order optimization works because it aligns the computational effort of optimization with the physical reality of the loss landscape: gradients provide high-frequency directional signals, while curvature provides low-frequency geometric context. Decoupling them yields the best of both worlds.

Key Foundational Work

SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent (2009) — contributed 1 concept elements to this direction.
BFGS with Update Skipping and Varying Memory (1998) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_periodic_curvature_estimation_9b20cef9,
  title = {periodic curvature estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/periodic-curvature-estimation/rdr_9b20cef9b7e74997b85514eefd0e1a6d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_periodic_curvature_estimation_9b20cef9_v1,
  title = {periodic curvature estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/periodic-curvature-estimation/rdr_9b20cef9b7e74997b85514eefd0e1a6d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake