Parallelizing Second-Order Optimization via Block-Diagonal Approximations

This direction is merged into another direction: Hessian dimensionality bottleneck. Its last version stays readable and citable here.

Version 1 of 112 papers17 conceptsfrozen 2026-10-04

Most deep learning models are trained using first-order methods like SGD or Adam. These algorithms are computationally "cheap" because they only look at the gradient—the slope of the loss landscape. However, they are blind to the curvature of that landscape. Second-order methods, such as Natural Gradient Descent or Newton’s method, use curvature information (stored in the Hessian or Fisher Information Matrix) to take more direct paths to the optimum, potentially converging in far fewer iterations.

The catch is the "curse of dimensionality." For a model with millions of parameters, the Hessian matrix contains trillions of entries. Storing and inverting such a matrix is impossible on modern hardware. To solve this, a growing body of research focuses on breaking this massive, global optimization problem into smaller, independent, and parallelizable sub-problems.

The Block-Diagonal Intuition

The central strategy for making second-order optimization practical is the block-diagonal approximation. Instead of treating every parameter's relationship with every other parameter, researchers assume that parameters in different layers (or even different parts of the same layer) are independent for the purpose of curvature.

Felix Dangel, Philipp Hennig, and Stefan Harmeling demonstrate in Modular Block-diagonal Curvature Approximations for Feedforward Architectures that you can partition large parameter matrices into smaller sub-blocks. This structural choice allows for significant parallelization with minimal impact on the quality of the optimization. By ignoring the "off-diagonal" correlations between distant layers, the math simplifies from one giant inversion to many small, simultaneous ones.

Decoupling Communication from Computation

Once the curvature matrix is structured as a series of independent blocks, the optimization task becomes "embarrassingly parallel." This allows for distributed architectures where the heavy lifting is spread across many processing units.

In Block-diagonal Hessian-free Optimization for Training Neural Networks, Caiming Xiong and colleagues show that by treating the curvature as block-diagonal, each parameter block can be updated using the Conjugate Gradient algorithm independently. This removes the need for inter-layer communication during the most intensive parts of the update, allowing distributed nodes to solve local optimization subproblems in isolation.

Similarly, Adnan Haider and Phil Woodland propose a master-worker framework in A Distributed Optimisation Framework Combining Natural Gradient with Hessian-Free for Discriminative Sequence Training. In this setup, workers process large data batches in parallel to gather second-order statistics, while a central process aggregates these to finalize the update. This decoupling ensures that the precision of second-order methods doesn't come at the cost of the scalability we expect from first-order methods.

Hardware-Aligned Approximations

To achieve "wall-clock" speeds competitive with SGD, the mathematical approximations must align with how GPUs and TPUs actually work. César Laurent and Pascal Vincent explore this in An Evaluation of Fisher Approximations Beyond Kronecker Factorization. They introduce approximations that transform sequential bottlenecks into batched operations, allowing the hardware to process multiple matrix inversions simultaneously.

This theme of "trading hardware for time" is also evident in meta-learning. Yilang Zhang and Georgios B. Giannakis show in Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation that complex meta-gradients can be expanded into independent terms. These terms allow for the parallel computation of multiple Hessian-vector products, effectively using available hardware parallelization to slash the sequential processing time of the meta-update.

Specialized Decompositions for Scale

Beyond general neural networks, specific problem domains benefit from unique decompositions:

By treating the curvature matrix not as a monolithic obstacle, but as a collection of independent modules, these researchers have turned second-order optimization from a theoretical luxury into a scalable tool for high-performance machine learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parallelizing_second_order_optimization__664eb66f,
  title = {Parallelizing Second-Order Optimization via Block-Diagonal Approximations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallelizing-second-order-optimization-via-block-diagonal-approximations/rdr_664eb66ff8054a71ada32dbf01393991},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_parallelizing_second_order_optimization__664eb66f_v1,
  title = {Parallelizing Second-Order Optimization via Block-Diagonal Approximations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallelizing-second-order-optimization-via-block-diagonal-approximations/rdr_664eb66ff8054a71ada32dbf01393991/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}