Kronecker Factored Curvature for Scalable Second Order Learning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2345 papers439 conceptsfrozen 2026-10-04

Standard deep learning relies almost exclusively on first-order optimization—following the gradient of the loss function. While efficient, this approach is "geometry-blind." It treats all parameter directions as equally sensitive, often leading to slow convergence in narrow valleys of the loss landscape or overconfident predictions.

Second-order methods, such as Natural Gradient Descent or the Laplace Approximation, account for the "curvature" of the landscape. By using the Hessian (the matrix of second derivatives) or the Fisher Information Matrix, these methods can warp the parameter space to make optimization more efficient and provide principled uncertainty estimates. However, for a model with $N$ parameters, these matrices contain $N^2$ elements. For a modern LLM, storing and inverting such a matrix is computationally impossible.

The research direction of Kronecker-factored Approximate Curvature (K-FAC) and its derivatives provides the mathematical bridge to make these second-order insights practical for large-scale neural networks.

The Intuition: Factoring the Landscape

The core insight of K-FAC, introduced by James Martens and Roger B. Grosse in Optimizing Neural Networks with Kronecker-factored Approximate Curvature, is that the curvature of a neural network layer has a highly specific structure.

Instead of treating the entire network's parameters as one giant vector, K-FAC assumes that the curvature is block-diagonal, where each block corresponds to a single layer. Crucially, it further approximates each layer's block as the Kronecker product of two much smaller matrices: one representing the covariance of the layer's inputs (activations) and another representing the covariance of the backpropagated gradients.

This factorization turns an intractable $O(N^2)$ problem into a series of small matrix operations. For a layer with $d_{in}$ inputs and $d_{out}$ outputs, the full curvature matrix is $(d_{in} \times d_{out})^2$. K-FAC reduces this to two matrices of size $d_{in}^2$ and $d_{out}^2$. This makes second-order updates only slightly more expensive than standard backpropagation while capturing the essential geometry of the weights.

From Optimization to Uncertainty

Once we have a scalable way to approximate the curvature, we can use it for more than just faster training. In Bayesian Deep Learning, the Laplace Approximation uses the curvature at a local minimum to "fit" a Gaussian distribution over the weights.

Erik A. Daxberger and Alexander Immer demonstrated in Laplace Redux - Effortless Bayesian Deep Learning that this classic method, when powered by K-FAC, allows standard pre-trained models to be converted into Bayesian models with almost no extra cost. This provides "introspection"—the ability for a model to know when it is likely to fail Bayesian Optimization Meets Laplace Approximation for Robotic Introspection.

This approach has recently scaled to the largest models in existence. Adam X. Yang and Maxime Robeyns introduced Laplace-LoRA Bayesian low-rank adaptation for large language models, which applies K-FAC to the low-rank adapters of LLMs like Llama-2. This significantly reduces overconfidence in language generation without the massive memory overhead of full-parameter Bayesian inference.

Auditing the Training Process

Kronecker-factored curvature also enables Data Attribution: the ability to ask, "Which specific training example caused the model to make this prediction?"

This is typically done via Influence Functions, which require inverting the Hessian matrix. By using K-FAC to approximate this inverse, researchers can now perform data attribution on massive generative models. For example, Sheng-Yu Wang and Aaron Hertzmann leveraged these approximations to identify training influences in diffusion models Fast Data Attribution for Text-to-Image Models. Similarly, Juhan Bae and Wu Lin used eigenvalue-corrected K-FAC to track how data influences the entire training trajectory Training Data Attribution via Approximate Unrolling.

Refining the Approximation

While the original K-FAC was a breakthrough, the field has moved toward more precise variants to handle the complexities of modern architectures:

By turning the intractable geometry of high-dimensional models into a structured, factorized representation, this research direction has transformed second-order information from a theoretical luxury into a practical tool for optimization, safety, and interpretability.

Papers in this version

Showing 60 of 345 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 239 papers
  2. Version 1 · 2026-10-04 · 345 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_kronecker_factored_curvature_approximati_a50872ce,
  title = {Kronecker-factored curvature approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kronecker-factored-curvature-approximation/rdr_a50872cef632437aa307aaddf116988c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_kronecker_factored_curvature_for_scalabl_a50872ce_v1,
  title = {Kronecker Factored Curvature for Scalable Second Order Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kronecker-factored-curvature-for-scalable-second-order-learning/rdr_a50872cef632437aa307aaddf116988c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}