Second-order optimization methods—such as Newton's method, Natural Gradient Descent, and Quasi-Newton updates—promise a direct path to the minimum of a loss function. By scaling the gradient step with the inverse Hessian matrix ($H^{-1}$), these algorithms automatically correct for ill-conditioned geometry, navigating steep valleys and flat plateaus with ease.
Yet, in deep learning, physical simulations, and multi-agent games, the raw Hessian often becomes a liability rather than an asset. In non-convex landscapes, the Hessian can develop negative eigenvalues, causing optimization steps to shoot upward or get trapped near saddle points. In high-dimensional or overparameterized regimes, the matrix frequently becomes rank-deficient or singular, leading to near-zero eigenvalues whose inverses blow up toward infinity, triggering catastrophic numerical explosions.
To bridge the gap between second-order theory and practical stability, a major research direction focuses on Hessian regularization and spectral stabilization. Rather than using the raw or naively approximated curvature, these techniques systematically reshape the eigenvalues of the Hessian to guarantee that it remains Symmetric Positive Definite (SPD) and well-conditioned, while preserving the essential geometric information required for rapid convergence.
Lifting the Floor: Diagonal Damping
The most intuitive way to stabilize an ill-conditioned Hessian is to establish a minimum curvature floor. If an eigenvalue $\lambda$ approaches zero, its inverse $1/\lambda$ approaches infinity. By adding a small diagonal perturbation—a technique known as damping or ridge regularization—the spectrum is shifted away from zero:
$$H_{\text{regulated}} = H + \lambda I$$
This simple transformation guarantees that the minimum eigenvalue is at least $\lambda$, bounding the maximum step size and ensuring invertibility. Lin et al. demonstrate that this diagonal shift is essential for preventing numerical failures in edge cases where optimization matrices become singular. Similarly, Huang utilizes a tuned perturbation term to maintain a well-defined update direction in high-dimensional stochastic environments.
Beyond standard optimization, diagonal damping is a critical tool for computing hyper-gradients in nested or hierarchical problems. In bilevel optimization and Stackelberg games, calculating the leader's gradient requires inverting the follower's Hessian. When the follower's landscape is flat or ill-conditioned, this inversion fails. To resolve this, Fiez et al. and Zheng introduce regularization terms directly to the implicit map, ensuring the inverse Hessian remains stable and well-defined during hierarchical gradient estimation. In optimal control, Xu et al. employ a proximal regularization technique that injects a small diagonal term to preserve the block-sparse structure of KKT systems while ensuring numerical robustness.
Flipping the Landscape: Absolute Value Filtering
While diagonal damping prevents eigenvalues from vanishing, it struggles in highly non-convex regions where the Hessian develops large negative eigenvalues. Simply adding a uniform identity matrix $\lambda I$ might not be enough to push these eigenvalues into positive territory unless $\lambda$ is chosen so large that it completely obliterates the true curvature information, reducing the algorithm to slow gradient descent.
To handle negative curvature without distorting the rest of the spectrum, researchers turn to spectral filtering. The most prominent variant is Absolute Value Filtering. Instead of shifting the eigenvalues, the Hessian is diagonalized, and any negative eigenvalue is replaced by its absolute value:
$$\lambda_i \leftarrow |\lambda_i|$$
Geometrically, this flips the local optimization landscape along the directions of negative curvature. A local maximum (a hill) or a saddle point is transformed into a local minimum (a bowl). This forces the optimizer to continuously descend rather than climbing up the landscape or stalling.
Shin et al. apply this absolute value trick to diagonal Hessian approximations, ensuring that the resulting preconditioner remains strictly positive and maintains a consistent descent direction near saddle points. In the domain of physical and computer graphics simulations, Chen et al. show that absolute eigenvalue filtering provides a parameter-free regularization strategy for Projected Newton solvers, removing the need to manually tune material-dependent clamping thresholds. Chen et al. unify this approach into a generalized trust-region framework, demonstrating that absolute value filtering acts as a conservative geometric constraint that stabilizes simulations during extreme deformations.
Targeted Corrections: Subspace and Structural Regularization
Modifying the entire diagonal or spectrum can sometimes introduce unwanted bias, regularizing directions that are already well-conditioned. A more precise alternative is to confine the regularization to specific mathematical subspaces or to enforce structural constraints directly on the curvature updates.
Row-Space Shifting
When a machine learning model achieves high confidence on a dataset, the objective function can become flat along directions orthogonal to the data, causing the Hessian to lose rank. To combat this, Kan et al. introduce a technique that shifts the Hessian matrix specifically within the row space of the data's linear model or Jacobian. By constraining the regularization to the span of the input features, optimization updates remain strictly bounded and aligned with relevant features, preventing the unbounded steps that typically occur in rank-deficient regions.
Trace and Determinant Constraints
Rather than modifying the matrix post-computation, regularization can be embedded directly into the algebraic properties of the update rules. For instance, Mueller introduces trace-based shrinkage damping, which continuously adjusts the curvature estimator toward its own trace. This prevents instability from near-zero eigenvalues on ill-conditioned loss surfaces more robustly than a static constant damping factor.
In the context of Quasi-Newton methods like BFGS, Curtis relaxes the strict requirement of satisfying the secant equation, opting instead to enforce geometric constraints on the trace and determinant of the Hessian approximation. This structural regularization allows the algorithm to naturally self-correct estimation errors over successive iterations in noisy, stochastic environments. Similarly, Mokhtari et al. incorporate a log-determinant penalty and an identity matrix term into decentralized BFGS updates, ensuring that local curvature matrices remain well-conditioned even when local nodes possess incomplete or highly ill-conditioned information.
Taming Approximations and Stale Curvature
In large-scale deep learning, computing the exact Hessian is computationally prohibitive. Practitioners rely on approximations—such as the empirical Fisher Information Matrix, randomized sketching, or "lazy" updates where the Hessian is computed infrequently. However, these approximations introduce structural errors and delays that can destabilize optimization if left unregularized.
When using "lazy" Hessians, the second-order information becomes increasingly outdated over successive iterations. Doikov et al. show that the resulting approximation error can be neutralized by dynamically scaling the regularization parameter in proportion to the update frequency. This mathematical balance allows the optimizer to safely use stale curvature data by taking more conservative steps as the data ages.
For randomized sketching methods, where the high-dimensional Hessian is projected into a lower-dimensional space, Zhang et al. demonstrate that incorporating a ridge term into the sketched matrix stabilizes the preconditioner. This shrinks the spectral radius of the optimization contour, making it more circular and preventing divergence when dealing with small sketch sizes or heavy-tailed data distributions.
Finally, in natural gradient descent, the empirical Fisher matrix is often used as a proxy for curvature. Zhang et al. reveal that proper scaling adjustments to this empirical approximation can render the optimizer highly robust to damping hyperparameters. This allows the system to remain stable even as the explicit damping values approach zero, simplifying the tuning process for deep architectures.
By treating curvature approximation not merely as a computational shortcut but as an implicit form of regularization, as argued by Semenov et al., modern optimization frameworks successfully exploit the speed of second-order geometry while maintaining the rock-solid stability of first-order methods.