Efficient Curvature Approximation for Neural Network Training

This direction is merged into another direction: curvature-aware optimization. Its last version stays readable and citable here.

Version 1 of 1201 papers208 conceptsfrozen 2026-10-04

Standard optimization in deep learning relies almost exclusively on first-order methods like SGD and Adam. These algorithms are computationally cheap, but they are "blind" to the local geometry of the loss landscape. When the optimization path enters a narrow, steep-sided valley—a condition known as ill-conditioning—first-order methods tend to oscillate wildly across the walls of the valley rather than sliding down its floor.

Second-order methods, which utilize the Hessian (the matrix of second derivatives) or the Fisher Information Matrix, solve this by accounting for the curvature of the landscape. By rescaling the gradient based on this curvature, they can take a direct path to the optimum. However, for a model with $N$ parameters, the Hessian has $N^2$ elements, making it impossible to store or invert for modern networks. The research direction explored here focuses on making these second-order insights practical through clever structural approximations.

Factoring the Curvature

The most successful family of approximations relies on the observation that neural network layers often have a Kronecker-factored structure. Instead of treating the entire Hessian as one giant block, researchers like James Martens (foundational to this area) and Yuhuai Wu have shown that we can approximate the Fisher Information Matrix as the Kronecker product of two much smaller matrices.

In An Empirical Analysis of Proximal Policy Optimization with Kronecker-factored Natural Gradients, Yuhuai Wu demonstrates that this approach, known as K-FAC, allows Reinforcement Learning agents to converge significantly faster by navigating the complex policy space more efficiently. By breaking the "optimization bottleneck," these methods reach target performance in fewer iterations, even if each iteration takes slightly longer to compute.

This structural insight has been extended to modern architectures beyond simple feed-forward networks. Runa Eschenhagen in Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures provides a unified framework for applying these approximations to Transformers and Graph Neural Networks, treating them as linear layers with weight-sharing to reduce the number of training steps by up to 50%.

Preconditioning as a Tensor Problem

While K-FAC focuses on the Fisher matrix, other researchers treat the parameters as multi-dimensional tensors. Vineet Gupta introduced Shampoo in Shampoo: Preconditioned Stochastic Tensor Optimization. Shampoo maintains separate preconditioners for each dimension of a parameter tensor. This "structure-aware" approach provides the benefits of second-order optimization—faster convergence and less hyperparameter tuning—while keeping the per-step runtime competitive with standard first-order methods.

Building on this, Nikhil Vyas and colleagues developed SOAP (SOAP: Improving and Stabilizing Shampoo using Adam). SOAP bridges the gap between Shampoo and Adam by running the Adam optimizer in the eigenbasis of the preconditioner. This hybrid approach achieves a 35% reduction in wall-clock time for language model pre-training, proving that second-order methods can outperform industry-standard first-order optimizers even in massive-scale settings.

Overcoming the Wall-Clock Barrier

The primary criticism of second-order methods is their high "per-step" cost. To address this, researchers have focused on hardware-aware implementations and hybrid strategies.

  1. Distributed Preconditioning: Shaohuai Shi and Wei Wang introduced DP-KFAC (Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning), which partitions the task of calculating and inverting Kronecker factors across multiple GPUs. This eliminates communication overhead and makes second-order methods practical for large-scale distributed training.
  2. Hardware-Driven Computation: Mengyun Chen developed THOR (THOR, Trace-based Hardware-driven Layer-Oriented Natural Gradient Descent Computation), which uses matrix traces to adaptively skip redundant updates. By matching the computation to the strengths of modern GPUs, THOR can train ResNet-50 on ImageNet in under three minutes.
  3. Subspace Integration: Rather than approximating the entire Hessian, Hadar Sivan proposed FOSI (FOSI: Hybrid First and Second Order Optimization). FOSI identifies the most "difficult" directions in the loss landscape (those with extreme curvature) and applies Newton’s method only to that small subspace, while using standard Adam for the rest. This targeted approach reduces training time to roughly 50-75% of the original duration.

Stability and Precision in Scientific Computing

Beyond standard deep learning, second-order methods are becoming essential for scientific applications where precision is paramount. In molecular dynamics, Siyu Hu developed RLEKF (RLEKF: An Optimizer for Deep Potential with Ab Initio Accuracy), which uses a reorganized Kalman filter to achieve 11x fewer training epochs than Adam while maintaining higher accuracy for energy and force predictions.

Similarly, for solving complex differential equations, Hao Zhang and Yongge Li found that quasi-Newton methods significantly outperform first-order methods in training physics-informed neural networks (Solving Fokker-Planck equation using deep learning).

Why It Works: The Intuition of Alignment

The fundamental reason these methods work is alignment. A first-order gradient points in the direction of steepest ascent, which is only the "best" direction if the loss landscape is a perfect circle. In the real world, landscapes are stretched and warped. Second-order methods use curvature information to "whiten" or "sphere" the landscape, ensuring that the update step is aligned with the true path to the minimum.

As Amirreza Valaei notes in the context of control theory (Second-Order Policy Gradient Methods for the Linear Quadratic Regulator), accounting for how a policy change shifts the underlying state distribution allows for quadratic convergence. By moving from "blind" steps to "curvature-aware" steps, these researchers are transforming optimization from a process of trial-and-error oscillation into a precise, direct descent.

Papers in this version

Showing 60 of 201 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_curvature_approximation_for_ne_3d40f965,
  title = {Efficient Curvature Approximation for Neural Network Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-curvature-approximation-for-neural-network-training/rdr_3d40f9655526461d896ff162aded7d2f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_curvature_approximation_for_ne_3d40f965_v1,
  title = {Efficient Curvature Approximation for Neural Network Training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-curvature-approximation-for-neural-network-training/rdr_3d40f9655526461d896ff162aded7d2f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}