First-order optimization methods like SGD and Adam dominate deep learning due to their simplicity and low computational overhead. However, they navigate the loss landscape blindly, ignoring its curvature. Second-order methods—such as Natural Gradient Descent (NGD), K-FAC, and Shampoo—promise a remedy by scaling gradients with the inverse of a curvature matrix (such as the Hessian or Fisher Information Matrix). In theory, this preconditions the step direction, allowing optimizers to glide through ill-conditioned landscapes, escape flat regions, and bypass saddle points much faster than first-order alternatives.
In practice, however, second-order methods are notoriously fragile. The core mathematical operation that gives them their power—matrix inversion ($G^{-1}$)—is also their Achilles' heel. When applied to deep neural networks, these curvature matrices frequently become ill-conditioned or nearly singular. Inverting an ill-conditioned matrix amplifies tiny numerical errors into catastrophic explosions, causing parameters to diverge.
This vulnerability is severely exacerbated by the modern shift toward low-precision arithmetic (such as FP16 and BF16), where limited dynamic range leads to underflow, rounding errors, and training crashes. A major research direction has emerged to resolve this, focusing on making second-order optimization numerically stable without sacrificing its convergence benefits.
The Anatomy of Numerical Failure
To understand why second-order methods fail, consider the standard preconditioned update rule:
$$\theta_{t+1} = \theta_t - \eta G^{-1} \nabla L(\theta_t)$$
where $G$ represents a curvature matrix. If $G$ has eigenvalues close to zero—which happens frequently near saddle points, poorly initialized regions, or when local inequality constraints are highly active—it becomes nearly singular. As noted by Cheng et al., standard gradient methods suffer from severe numerical instability during these inversion steps when the matrix lacks well-conditioned properties. In a distributed setting, Kozma et al. point out that highly active constraints can cause a "dual Hessian singularity," causing standard linear solvers to fail entirely.
[Curvature Matrix G] ---> Near-Zero Eigenvalues (Ill-Conditioning)
|
v
[Matrix Inversion G⁻¹] ---> Amplifies Numerical Noise / Errors
|
v
[Low-Precision (FP16)] ---> Underflow to Zero / Overflow to NaN
|
v
Catastrophic Divergence
This problem transitions from problematic to fatal when moving from 32-bit to 16-bit floating-point representations. As Bae et al. and Eschenhagen et al. observe, second-order methods frequently fail or require excessive damping when applied in low-precision training regimes. In 16-bit half-precision, small values underflow to zero, making matrices artificially singular, while large values overflow, leading to NaN errors and training crashes (Hong).
Furthermore, advanced optimizers often rely on computing inverse matrix roots (like $G^{-1/2}$), which Mei et al. show are highly sensitive to numerical errors in standard precision when matrices are ill-conditioned. Even when computational shortcuts like caching matrix inversions are employed to reduce overhead, maintaining stability remains a significant challenge (Frans et al.). In distributed or noisy environments, process and observation noise can corrupt these curvature approximations further, causing the pre-conditioning matrices to become non-invertible and the optimization process to diverge (Chakrabarti et al.).
Bypassing Inversion Entirely
The most direct solution to this dilemma is to eliminate the matrix inversion step altogether. If an algorithm never explicitly computes $G^{-1}$, it cannot suffer from the numerical explosions triggered by inversion.
One prominent strategy is to perform optimization in an alternative mathematical space where curvature updates can be computed natively without inversion. For instance, Lin et al. (2024) introduced Structured Inverse-Free Natural Gradient Descent (SINGD), which maps updates into a "matrix logarithm space." By operating in the log-domain, the algorithm bypasses unstable inversions entirely, maintaining numerical stability even in BFP16 precision.
Similarly, Lin et al. (2023) proposed Generalized Normal Coordinates (GNCs) to simplify optimization over the manifold of symmetric positive-definite (SPD) matrices. By locally "flattening" the curved manifold so it resembles standard Euclidean space, the authors enable efficient, matrix-inverse-free momentum updates that sidestep the low-precision instability of traditional K-FAC.
Another angle is to directly track and update the inverse of the preconditioner over time, rather than recomputing the matrix and inverting it at each step. This can be achieved via Quasi-Newton approximations to avoid direct inversion (Luo et al.). Alternatively, Mei et al. leverage this in KrADagrad, using a Kronecker Approximation-Domination factorization to update the inverse preconditioner directly, avoiding precision-sensitive matrix root inversions on standard 32-bit hardware.
Load-Bearing Heuristics and Surrogates
When explicit curvature approximations are retained, researchers rely on stabilizing heuristics to prevent divergence. The most common tool is damping—adding a small identity matrix scalar ($\lambda I$) to the curvature matrix before inversion to bound its eigenvalues away from zero.
Clarke et al. demonstrate that these second-order heuristics, such as automatic learning rate selection and damping, are so critical that grafting them onto first-order methods (as in AdamQLR) provides substantial stability and performance boosts without manual tuning. However, relying purely on static damping can be insufficient when initialized far from the optimum.
To guarantee global convergence without expensive line searches, Liu et al. utilize quadratic and cubic surrogate functions that prevent optimization "blow-ups" or stalling in regions where derivatives vanish. In distributed environments, Kozma et al. prevent dual Hessian singularities by introducing an $L_2$ constraint relaxation strategy.
Taking a dynamical systems perspective, Huh analyzes the learning trajectories of deep networks to show that curvature correction primarily alters the velocity rather than the path of optimization. The author proposes "Fractional NGD" ($\sqrt{\text{NGD}}$), a middle-ground approach that tempers standard second-order acceleration to capture its benefits while avoiding the overfitting and instability typically found near saddle points.
Structural Approximations and Low-Rank Updates
A final frontier in stabilizing second-order methods involves changing how the curvature matrices are structured and updated. Computing and caching full matrix inversions introduces significant overhead and vulnerability to accumulated computational errors (Frans et al.). To mitigate this, modern methods break large curvature matrices down into smaller, more manageable Kronecker factors.
However, even these factors can suffer from instability during half-precision arithmetic (Mozaffari et al.). To solve this, Mozaffari et al. developed MKOR, which uses fast rank-1 updates to modify the Kronecker factors, significantly slashing computational costs while avoiding complex, error-prone matrix decompositions.
By combining structural constraints with normalization techniques—such as the instant-sign normalization and iterate averaging found in the SPlus optimizer (Frans et al.)—recent architectures can safely exploit curvature information at scale, bridging the gap between the theoretical speed of second-order optimization and the practical constraints of low-precision hardware.
Key Foundational Work
Curvature-corrected learning dynamics in deep neural networks (2020) — contributed 1 concept elements to this direction.
An improved Distributed Dual Newton-CG method for convex quadratic programming problems (2014) — contributed 1 concept elements to this direction.