Second-order optimization methods—such as Natural Gradient Descent, KFAC, and Bundle Adjustment—offer a compelling promise: by leveraging the curvature of the optimization landscape, they can find optimal solutions in far fewer iterations than standard first-order gradient descent. To do this, these methods scale the gradient using a preconditioning matrix, which typically requires computing and inverting a curvature matrix like the Hessian or the Fisher Information Matrix.
However, a major bottleneck prevents these methods from dominating modern machine learning and large-scale engineering: numerical instability. As hardware shifts toward lower-precision arithmetic (such as FP32 for geometric reconstruction, or FP16 and BF16 for deep learning), the explicit inversion of these curvature matrices frequently leads to catastrophic numerical failure, underflow, or divergence.
A growing body of research addresses this bottleneck by developing inversion-free and square-root preconditioning methods. The core intuition behind this paradigm shift is simple: we can retain the acceleration benefits of second-order curvature without ever explicitly squaring a condition number or computing a matrix inversion.
The Condition Number Trap
To understand why traditional second-order methods fail in low precision, we must look at how curvature matrices are constructed. In many optimization problems, the local curvature is represented by a Hessian matrix $H$, which is often approximated as the product of Jacobians:
$$H = J^T J$$
If the Jacobian matrix $J$ has a condition number $\kappa$, the condition number of the Hessian $H$ becomes $\kappa^2$.
In floating-point arithmetic, the precision budget is strictly limited. When you square the condition number, the dynamic range required to represent the matrix accurately doubles. Small singular values that were perfectly resolvable in $J$ suddenly underflow or become indistinguishable from numerical noise in $H$. When we attempt to invert this ill-conditioned Hessian ($H^{-1}$), these noisy, tiny values are amplified into massive errors, causing the optimizer to diverge.
Bypassing the Square via QR Decomposition
The first core mechanism for avoiding this instability is to operate directly on the Jacobian matrix rather than the squared Hessian.
In large-scale 3D reconstruction and computer vision, the standard approach to solving bundle adjustment problems has long relied on the Schur complement trick applied to the Hessian. To circumvent the resulting numerical degradation, Nikolaus Demmel et al. introduced a formulation known as Square Root Bundle Adjustment in Square Root Bundle Adjustment for Large-Scale Reconstruction.
Instead of forming $H = J^T J$, the authors utilize a specialized QR decomposition directly on the Jacobian matrix $J$:
$$J = Q R$$
Because the upper triangular matrix $R$ preserves the exact condition number $\kappa$ of the Jacobian (rather than $\kappa^2$), the linear system can be solved with far greater numerical fidelity. In a follow-up work, Square Root Marginalization for Sliding-Window Bundle Adjustment, Nikolaus Demmel et al. extended this concept to sliding-window visual-inertial odometry, replacing traditional Hessian-based marginalization with QR-based nullspace marginalization.
By keeping the condition number linear rather than quadratic, these square-root formulations allow single-precision (32-bit) floating-point arithmetic to achieve the exact same accuracy as standard double-precision (64-bit) solvers, while cutting computational overhead in half.
Mapping Inversion to Subtraction in Log Space
In deep learning, where models are routinely trained in 16-bit formats like BF16, avoiding the condition number trap requires a different strategy. We cannot easily run a full QR decomposition on the massive, dynamically changing weight matrices of a neural network. Instead, we must find a way to update the preconditioner without explicit inversion.
One elegant solution is to perform optimization updates in an entirely different mathematical space. In Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC, Wu Lin et al. proposed an algorithm called SINGD that maps curvature factors into a matrix logarithm space.
The intuition relies on a fundamental property of logarithms. For scalars, $\log(x^{-1}) = -\log(x)$. For a symmetric positive-definite matrix $A$, a similar relationship holds in matrix logarithm space:
$$\log(A^{-1}) = -\log(A)$$
By parameterizing and updating the curvature factors directly in this log-space, the operation of matrix inversion is transformed from a dangerous, division-like operation into a perfectly stable, linear operation: additive subtraction. When it is time to apply the preconditioner, the optimizer projects the log-space factors back via matrix exponentiation. This algebraic transformation completely eliminates the division-by-zero style explosions that typically cause second-order methods to crash in 16-bit floating-point formats.
Direct Inverse Parameterization
Another way to avoid explicit matrix inversion is to maintain and update an approximation of the inverse curvature matrix directly throughout training, rather than maintaining the curvature matrix itself and inverting it at each step.
In KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization, Jonathan Mei et al. designed an optimizer that bypasses the precision-sensitive matrix inversions seen in methods like Shampoo. Instead of accumulating curvature and then computing inverse roots, KrADagrad uses the Woodbury matrix identity to apply rank-updates directly to the inverse of the preconditioner factors. Furthermore, the authors demonstrate that utilizing positive matrix roots instead of inverse roots allows curvature-aware training to achieve stable convergence on standard 32-bit hardware without needing 64-bit emulation. This proves that the precision bottleneck in second-order deep learning is not a fundamental trait of curvature tracking, but rather an artifact of how the matrices are updated.
Taking this concept to its logical conclusion, Wu Lin et al. explored whether the square-root operation itself could be abandoned entirely. In Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective, the authors showed that by exploiting preconditioner invariance, matrix-based optimizers can be structured to update their states using only basic matrix multiplications. This eliminates both square roots and inversions, unlocking stable second-order preconditioning in low-precision environments.
Amortization and Submanifold Geometry
A final perspective on inversion-free optimization frames the problem geometrically or variationally. Instead of solving an algebraic equation to find an inverse matrix, can we find the preconditioned step by solving a simpler optimization problem?
In Amortized Proximal Optimization, Juhan Bae et al. demonstrated that structured preconditioning matrices can be learned directly by optimizing a proximal objective. This formulation entirely bypasses both explicit matrix inversions and eigendecompositions, amortizing the cost of preconditioning and ensuring numerical robustness in low-precision training environments.
Similarly, we can view the space of symmetric positive-definite (SPD) preconditioning matrices as a curved geometric surface—a Riemannian submanifold. In Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning, Wu Lin et al. introduced Generalized Normal Coordinates (GNCs) to locally "flatten" this manifold. By mapping the curved space of SPD matrices into a local Euclidean coordinate system, the authors enabled momentum updates that respect the underlying Lie group structure of the matrices. Because the geometry is handled natively by the coordinate system, the optimizer executes second-order updates smoothly without ever needing to invert a large Fisher Information Matrix.
| Method / Paradigm | Core Mechanism | Primary Benefit | Target Domain |
|---|---|---|---|
| Square-Root $\sqrt{BA}$ (Demmel et al.) | QR decomposition on the Jacobian matrix directly | Keeps condition number linear ($\kappa$ vs $\kappa^2$); enables FP32 to match DP64 | Large-Scale 3D Reconstruction / VIO |
| Log-Space KFAC (SINGD) (Lin et al.) | Parameterizes curvature factors in matrix log-space | Converts multiplicative inversion into stable additive subtraction; prevents BF16 crashes | Deep Learning (Transformers / CNNs) |
| Direct Inverse Tracking (Mei et al.) | Woodbury identity rank-updates on inverse factors | Bypasses explicit inversion steps entirely on standard hardware | Curvature-Aware Stochastic Optimization |
| Submanifold / Proximal Learning (Bae et al., Lin et al.) | Amortized proximal objectives & Generalized Normal Coordinates | Eliminates algebraic inversion by optimizing directly on the SPD manifold | Low-Precision Deep Learning |
Summary
The traditional view that second-order optimization requires expensive, fragile matrix inversions is giving way to a more robust paradigm. Whether by preserving condition numbers through QR decompositions, mapping inversion to subtraction via matrix logarithms, or tracking inverse states directly through algebraic identities, inversion-free preconditioning bridges the gap between second-order mathematical efficiency and low-precision hardware execution.
Key Foundational Work
Amortized Proximal Optimization (2022) — contributed 2 concept elements to this direction.
Square Root Bundle Adjustment for Large-Scale Reconstruction (2021) — contributed 2 concept elements to this direction.
Square Root Marginalization for Sliding-Window Bundle Adjustment (2021) — contributed 1 concept elements to this direction.