First-order optimization methods like Stochastic Gradient Descent (SGD) and Adam navigate the loss landscape using only the local slope (the gradient). They require practitioners to tune relatively few parameters—primarily the learning rate. In contrast, second-order methods (such as Newton's method, Natural Gradient Descent, and Quasi-Newton approximations like L-BFGS) leverage the curvature of the loss landscape by calculating or approximating the Hessian or Fisher Information Matrix.
In theory, knowing the curvature allows an optimizer to take a direct path to the minimum, bypassing the agonizingly slow progress characteristic of ill-conditioned, high-curvature valleys. Yet, in practice, deep learning remains dominated by first-order methods.
The barrier to widespread adoption is a hidden tax: second-order methods introduce a significant explosion in hyperparameter complexity and sensitivity. As observed by Vineet Gupta et al. in their work on Shampoo, these algorithms exhibit a far greater sensitivity to hyperparameter tuning than robust first-order alternatives, frequently leading to instability or suboptimal performance if every parameter is not precisely calibrated.
The Inversion Problem and the Necessity of Damping
To understand why second-order methods introduce so many sensitive hyperparameters, we must look at the core mathematical operation they perform: inverting a curvature matrix. A standard Newton-style step updates parameters $\theta$ by multiplying the gradient $\nabla L(\theta)$ by the inverse of the Hessian matrix $H$:
$$\Delta \theta = - H^{-1} \nabla L(\theta)$$
In deep learning, the loss landscape is highly non-convex and noisy. Consequently, the curvature matrix often contains eigenvalues that are negative or extremely close to zero. When you invert a matrix with an eigenvalue near zero, the inverse explodes toward infinity. In an optimization loop, this manifests as a catastrophic step that flings the model parameters into regions of poor generalization or numerical divergence.
To prevent these explosions, second-order algorithms rely on damping. Damping adds a scalar value $\lambda$ to the diagonal of the matrix before inversion, effectively penalizing overly aggressive steps:
$$\Delta \theta = - (H + \lambda I)^{-1} \nabla L(\theta)$$
However, selecting the value of $\lambda$ introduces a severe tuning dilemma. As Chao Zhang et al. demonstrate in their analysis of the Improved Empirical Fisher, second-order methods are highly sensitive to these damping parameters.
If $\lambda$ is chosen too large, it dominates the curvature matrix, washing out the second-order geometric information and causing the algorithm to behave like standard, slow gradient descent. If $\lambda$ is too small, the matrix remains ill-conditioned, leading to numerical instability.
Alberto Bernacchia et al. point out in Exact natural gradient in deep linear networks and its application to the nonlinear case that there is currently no principled, automated method for determining these damping factors across different neural network architectures. This sensitivity is so pervasive that it even impacts downstream tasks; Anshuman Suri et al. found that when utilizing second-order derivatives for privacy auditing, the lack of a universal heuristic for tuning damping factors to manage near-zero eigenvalues leads to highly inconsistent results (Do Parameters Reveal More than Loss for Membership Inference?).
The Computational Compromise of Lazy Updates
Computing and inverting a massive curvature matrix at every single training step is computationally prohibitive for modern neural networks. To make second-order optimization practical, researchers must use approximations and "lazy" updates—reusing a computed matrix for several iterations while only updating the gradient.
While this strategy drastically reduces the computational cost per iteration, it introduces a new, highly sensitive hyperparameter: the update frequency.
Standard Second-Order: Compute Curvature ──> Invert Matrix ──> Take Step (Every Iteration)
Lazy Second-Order: Compute Curvature ──> Invert Matrix ──> Take Step ──> Take Step (Stale Matrix)
└─────────────────── New Hyperparameter: Interval ───────────────────┘
Practitioners must now decide exactly how many steps a curvature matrix can be safely reused before it becomes dangerously inaccurate. Nikita Doikov et al. formalized this trade-off in Second-Order Optimization with Lazy Hessians, showing that while laziness can preserve fast convergence rates, it requires balancing regularization parameters that are tightly coupled with the landscape's geometry.
This update-interval hyperparameter appears across nearly all modern structural approximations:
- Kronecker Factorization: In methods like KrADagrad, Jonathan Mei et al. note that managing the frequency of preconditioner updates increases the overall complexity of model tuning.
- Mini-Block Approximations: Achraf Bahamou et al. highlight that the frequency of curvature statistic updates introduces a delicate sensitivity balance in their Mini-Block Fisher framework.
- Neural Fields: Even in specialized domains, Shin-Fang Ch'ng et al. observe that the optimal interval between preconditioner updates varies significantly across different scenes, necessitating manual, scene-by-scene tuning (Preconditioners for the Stochastic Training of Neural Fields).
Every computational shortcut taken to make second-order methods viable creates another hyperparameter knob that must be manually adjusted.
Trust Regions and Regularization Complexity
When moving beyond simple damping, advanced second-order algorithms attempt to stabilize training by constraining the optimization step within a "trust region" or by applying higher-order regularization. These mathematical frameworks introduce abstract parameters that lack intuitive physical meanings, making them notoriously difficult to tune.
For instance, cubic regularization methods modify the local Taylor expansion by adding a cubic penalty term proportional to the step size cubed. The weight of this penalty is directly tied to the Lipschitz constant of the Hessian matrix. As Ruichen Jiang et al. point out in Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate, this Lipschitz constant is virtually impossible to estimate a priori for real-world deep networks, leaving practitioners to rely on complex backtracking strategies or manual guesswork.
Similarly, trust-region methods restrict the step size to a local neighborhood where the quadratic approximation of the loss surface is assumed to be accurate. In algorithms like ACKTR, developed by Yuhuai Wu et al., the user must tune trust-region bounds alongside damping coefficients (Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation). If these bounds are not precisely calibrated for the specific environment or architecture, performance becomes highly inconsistent.
Why Universal Heuristics Fail
First-order optimizers like Adam owe their ubiquity to robust default hyperparameters (such as $\beta_1 = 0.9$ and $\beta_2 = 0.999$) that perform reliably across a vast spectrum of architectures and datasets. Second-order hyperparameters enjoy no such luxury because they are intrinsically coupled with the local, changing geometry of the loss landscape.
In continuous dynamical systems, Vassilis Apidopoulos demonstrates that determining optimal damping parameters requires precise knowledge of the objective function's curvature and growth constants—values that are completely unknown and dynamically shifting during deep network training (Convergence rates for the Heavy-Ball continuous dynamics for non-convex optimization, under Polyak-Łojasiewicz condition).
Furthermore, as Xi-Lin Li notes in Preconditioner on Matrix Lie Group for SGD, even when second-order preconditioners are elegantly structured on matrix Lie groups to improve convergence, they still exhibit a stubborn sensitivity to damping factors and learning rates that demands architecture-specific tuning.
Ultimately, while second-order optimization offers a mathematically superior trajectory on paper, it shifts the engineering burden from managing optimization time to managing hyperparameter search budgets. Until the field develops automated, robust mechanisms to dynamically tune damping factors, update frequencies, and regularization constants, second-order methods will remain a powerful but finicky alternative to the plug-and-play simplicity of first-order gradients.
Key Foundational Work
Topmoumoute Online Natural Gradient Algorithm (2007) — contributed 1 concept elements to this direction.