Inverse-Free Second-Order Optimization for Low-Precision Training

This direction is split into other directions: inversion-free preconditioning. Its last version stays readable and citable here.

Version 1 of 111 papers21 conceptsfrozen 2026-10-04

Second-order optimizers, such as Natural Gradient Descent and KFAC, promise faster convergence by accounting for the curvature of the loss landscape. However, they have historically struggled with numerical instability, particularly when implemented in the low-precision floating-point formats (like FP16 or BF16) required for modern large-scale training.

The core of this instability lies in the matrix inversion or inverse square root operations required to compute a preconditioner. When a curvature matrix is ill-conditioned—meaning some of its eigenvalues are very close to zero—inverting it produces massive values that lead to divergent training. Recent research has converged on a shared solution: restructuring optimization to be "inverse-free," allowing curvature-aware updates to remain stable without the overhead of high-precision arithmetic.

The Precision Bottleneck in Curvature Awareness

In standard second-order methods, we update parameters by multiplying the gradient by the inverse of a preconditioner (like the Fisher Information Matrix). In low-precision environments, the limited dynamic range of 16-bit floats cannot represent the extremely large values generated by inverting small eigenvalues.

Jonathan Mei, Alexander Moreno, and Luke Walters argue in KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization that this precision bottleneck is an algorithmic artifact rather than a fundamental requirement. They demonstrate that by using a "Kronecker Approximation-Domination" (KrAD) factorization and the Woodbury matrix identity, one can update the inverse of the preconditioner directly. By bypassing the explicit inversion step, they achieve performance parity between 32-bit and 64-bit implementations, proving that second-order methods can be robust on standard hardware.

Moving to Matrix Logarithm Space

One of the most effective ways to eliminate inversion is to change the mathematical space in which the optimizer operates. In Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC, Wu Lin, Felix Dangel, and their colleagues introduce SINGD.

The intuition behind SINGD is to perform curvature updates in a matrix logarithm space. In this space, the multiplicative operation of matrix inversion is transformed into a simple additive subtraction. This transformation is naturally more stable; while dividing by a tiny number in FP16 causes an overflow, subtracting a value in log-space remains well-behaved. This allows SINGD to maintain the benefits of KFAC while remaining stable in BFP16 and as memory-efficient as AdamW.

Flattening the Manifold

Another perspective on this problem comes from Riemannian geometry. Symmetric positive-definite (SPD) matrices, which represent curvature, live on a curved manifold. Standard optimization steps can easily "fall off" this manifold, leading to non-sensical results.

In Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning, Wu Lin, Valentin Duruisseaux, and Mark Schmidt introduce Generalized Normal Coordinates (GNCs). GNCs effectively "flatten" the curved manifold locally so it looks like standard Euclidean space. This allows for momentum updates that are entirely inverse-free. By utilizing Lie group structures and algebraic symmetries, the authors avoid the numerical errors associated with inverting the Fisher Information Matrix, making the optimizer significantly more robust for deep learning tasks in low-precision formats.

Removing the Square Root

The drive toward stability also extends to first-order adaptive methods like Adam, which rely on the square root of second-moment estimates. In Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective, Wu Lin and Alireza Makhzani explore removing both square roots and inversions.

They leverage preconditioner invariance, a property that allows the optimizer state to be updated through simple matrix multiplications. This approach prevents the amplification of quantization errors. Similarly, Difan Zou and Jichu Li show in A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization that using SVD-based sign operators for parameter updates provides superior stability. By avoiding division by small estimates, these mechanisms ensure that quantization errors do not explode during the update step.

Summary of the Shift

The transition from traditional second-order methods to these modern variants represents a fundamental shift in how we handle curvature:

  1. From Inversion to Addition: Using matrix logarithms (SINGD) or direct inverse updates (KrADagrad) to avoid the "division by zero" problem in low precision.
  2. From Curved to Flat: Using coordinate transformations (GNC) to simplify the geometry of the optimization space.
  3. From Roots to Invariance: Removing square root operations to simplify the computational graph and reduce sensitivity to small values.

By treating numerical stability as a first-class design constraint, these researchers have made sophisticated curvature-aware optimization practical for the current era of low-precision, large-scale deep learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_inverse_free_second_order_optimization_f_bc79328a,
  title = {Inverse-Free Second-Order Optimization for Low-Precision Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inverse-free-second-order-optimization-for-low-precision-training/rdr_bc79328a051e40a481cc2135a5bdb9db},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_inverse_free_second_order_optimization_f_bc79328a_v1,
  title = {Inverse-Free Second-Order Optimization for Low-Precision Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inverse-free-second-order-optimization-for-low-precision-training/rdr_bc79328a051e40a481cc2135a5bdb9db/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}