Scalable Second-Order Optimization for Neural Networks

This direction is split into other directions: Hessian computation bottleneck. Its last version stays readable and citable here.

Version 1 of 11,244 papers1,360 conceptsfrozen 2026-10-04

In the landscape of deep learning, first-order optimizers like Stochastic Gradient Descent (SGD) and Adam are the standard. They are computationally "cheap," requiring only the gradient—a vector that points in the direction of steepest ascent. However, these methods are effectively "blind" to the local geometry of the loss landscape. They don't know if they are in a narrow canyon or a wide-open plain, often leading to slow convergence or getting trapped in the "saddle points" prevalent in high-dimensional non-convex spaces.

Second-order methods, such as Newton’s method, solve this by using the Hessian matrix—a map of the landscape's curvature. While a gradient tells you which way is down, the Hessian tells you how the slope is changing. This allows for much larger, more intelligent steps. The catch? For a model with $d$ parameters, the Hessian has $d^2$ entries. For a modern neural network with millions of parameters, storing and inverting this matrix is computationally impossible.

The research direction explored here focuses on making these second-order insights tractable, bridging the gap between the efficiency of SGD and the mathematical power of Newton’s method.

The Geometry of the Bottleneck

The primary obstacle is scaling. As noted by Naman Agarwal, Brian Bullins, and Tengyu Ma in Finding Local Minima for Nonconvex Optimization in Linear Time, traditional second-order methods scale cubically with dimensionality. In a world of billion-parameter models, we cannot even store the Hessian, let alone invert it.

To bypass this, researchers focus on Hessian-free or Quasi-Newton approaches. Instead of calculating the full matrix, these methods approximate the effect of the Hessian. A key tool is the Hessian-vector product (HVP). It turns out we can compute the product of the Hessian and a vector $Hv$ with roughly the same computational cost as a standard gradient. Ross M Clarke and colleagues in Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation demonstrate that by using an infinite series of these products, one can approximate the inverse Hessian's action without ever materializing the matrix itself.

Squeezing Curvature from Noisy Data

In deep learning, we rarely have the "true" gradient; we have a noisy estimate from a mini-batch. Second-order methods are notoriously sensitive to this noise. If your estimate of the curvature is wrong, your "intelligent" step might launch you into a region of high loss.

To solve this, researchers like Donald Goldfarb and Peter Richtárik developed Stochastic Block BFGS. This method uses "sketches"—random low-dimensional projections of the Hessian—to capture the most important curvature information while ignoring the noise.

Another strategy is Variance Reduction. Aurelien Lucchi and Thomas Hofmann show in A Variance Reduced Stochastic Newton Method that by combining second-order updates with techniques like SVRG (Stochastic Variance Reduced Gradient), we can stabilize the curvature estimates. This allows the optimizer to maintain the fast, quadratic convergence of Newton's method even when training on small, noisy slices of data.

One of the greatest strengths of second-order information is its ability to handle saddle points. In high dimensions, most points with a zero gradient aren't local minima; they are saddles—valleys in one direction and peaks in another. First-order methods often crawl to a halt near these points.

Second-order methods can "see" the directions of negative curvature. Fred Roosta and Peng Xu argue in Second-Order Optimization for Non-Convex Machine Learning that methods like Trust Region and Cubic Regularization are uniquely capable of "teleporting" out of these traps. By incorporating momentum into these cubic methods, as explored by El Mahdi Chayti and Martin Jaggi in Improving Stochastic Cubic Newton with Momentum, researchers are creating optimizers that are both globally convergent and robust to the chaotic landscapes of deep networks.

Beyond Training: Influence and Uncertainty

The utility of scalable curvature extends beyond just updating weights. The Hessian represents the model's sensitivity to its parameters, which is vital for understanding data influence and uncertainty.

  • Influence Estimation: To find which training samples most influenced a prediction, one traditionally needs to invert the Hessian. Xinyu Zhou and Martin Jaggi introduced HyperINF, which uses iterative methods to approximate this inversion, making it possible to audit foundation models.
  • Model Selection: Alexander Immer and colleagues demonstrate in Scalable Marginal Likelihood Estimation that Laplace approximations—which rely on the Hessian—can be used for hyperparameter tuning and out-of-distribution detection without the need for expensive cross-validation.

By treating the Hessian not as a monolithic matrix to be inverted, but as a dynamic operator to be approximated, this field is turning second-order optimization from a theoretical ideal into a practical engine for large-scale machine learning.

Papers in this version

Showing 60 of 1,244 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scalable_second_order_optimization_for_n_2c94b353,
  title = {Scalable Second-Order Optimization for Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-second-order-optimization-for-neural-networks/rdr_2c94b35347cf473a89ff61280d0dbc96},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_scalable_second_order_optimization_for_n_2c94b353_v1,
  title = {Scalable Second-Order Optimization for Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-second-order-optimization-for-neural-networks/rdr_2c94b35347cf473a89ff61280d0dbc96/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}