In the landscape of deep learning, first-order optimizers like SGD and Adam are the standard because they are computationally "cheap." Each step requires only a gradient, which is easily obtained via backpropagation. Second-order methods, which utilize the curvature of the loss landscape (the Hessian matrix), offer a tantalizing alternative: they can converge in far fewer iterations by taking more direct paths to the minimum.
However, a persistent challenge prevents their widespread adoption: per-iteration computational overhead. While a second-order method might require 10x fewer iterations to converge, each of those iterations might be 100x slower than a first-order step. As Helmbold et al. observed, the high per-iteration costs of operations like Hessian matrix inversion often make these methods slower in terms of total wall-clock time.
The Sources of Computational Complexity
The overhead in second-order optimization stems from three primary mathematical requirements:
- Curvature Estimation and Inversion: Calculating the exact Hessian matrix for a model with millions of parameters is impossible. Even calculating approximations like the Fisher Information Matrix (FIM) or using Kronecker-Factored Approximate Curvature (K-FAC) introduces significant costs. Osawa et al. note that the heavy lifting of calculating and inverting these matrices typically outweighs the benefits of reduced training steps.
- Solving Subproblems: Many advanced methods, such as Cubic Regularization or Trust-Region methods, require solving an auxiliary subproblem at every step to find the optimal update. Wang et al. and Mokhtari et al. highlight that solving these cubic or quadratic subproblems remains expensive even with efficient solvers like Lanczos methods.
- Hessian-Vector Products: Methods that avoid explicit matrix construction often rely on Hessian-vector products (HVPs). While more efficient, Zeng et al. and Rathore et al. point out that these products still impose a significant burden, often making iterations several to hundreds of times slower than standard backpropagation.
The Wall-Clock Time Trade-off
The central goal of recent research is to ensure that the reduction in total iterations actually translates to a reduction in wall-clock time. Yong et al. quantify this by showing that even efficient preconditioned optimizers often incur a 10% to 25% additional overhead per step, which can be a dealbreaker in resource-constrained environments.
This overhead is particularly acute in specialized domains. In Physics-Informed Neural Networks (PINNs), where the loss function involves complex differential operators, the cost of second-order information scales poorly with the complexity of the physics being modeled Rathore et al.. Similarly, in Generative Adversarial Networks (GANs), the use of mixed Hessians to stabilize competitive training creates a practical barrier for large-scale deployment Schäfer 0001 et al..
Strategies for Mitigation
To make second-order methods competitive, researchers have developed several strategies to "hide" or reduce the per-iteration cost:
Pipelining and Bubble Filling
One of the most innovative approaches involves distributed systems. Osawa et al. introduced PipeFisher, which identifies "pipeline bubbles"—idle periods where GPUs wait for data in distributed training—and fills them with second-order curvature calculations. By utilizing otherwise wasted cycles, they achieved near-perfect GPU utilization and reduced total training time.
Amortization and Frequency Tuning
Rather than computing curvature at every step, many methods update the Hessian information periodically. Mueller et al. suggest that amortizing these updates over multiple steps can significantly reduce average costs. However, this introduces a new challenge: finding the optimal frequency to balance stability and speed Liu et al..
Low-Rank and Quasi-Newton Approximations
Instead of full second-order information, researchers use approximations that scale more gracefully. Huang et al. proposed SPAN, which uses randomized low-rank approximations to achieve a better trade-off between per-iteration speed and global convergence. Similarly, Mokhtari et al. developed decentralized quasi-Newton methods that approximate curvature locally to avoid the communication overhead of sharing massive matrices across a network.
Hybrid Heuristics
Finally, some researchers attempt to graft second-order benefits onto first-order frameworks. Clarke et al. explored AdamQLR, which applies second-order heuristics like automatic damping and learning rate selection to the Adam optimizer. While this increases the cost per step due to extra forward/backward passes, the authors argue the improved stability can reduce the need for expensive hyperparameter tuning.
By addressing the per-iteration bottleneck, these works aim to transform second-order optimization from a theoretical curiosity into a practical tool for training the next generation of large-scale models.