Scalable Fisher Information for Deep Learning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2218 papers234 conceptsfrozen 2026-10-04

When we train a neural network, we usually imagine ourselves descending a landscape where the "steepness" is defined by the loss function. Standard Stochastic Gradient Descent (SGD) treats this landscape as flat and Euclidean—a step of size $\epsilon$ in parameter space is assumed to have a consistent effect.

However, as Shun-ichi Amari pointed out in his seminal work on Natural Riemannian Gradients, the parameter space of a neural network is not flat. It is a "neuromanifold" where the distance between two models should be measured by how much their output distributions differ, not just their raw weights. The mathematical tool that captures this internal geometry is the Fisher Information Matrix (FIM).

The Geometry of Information

The Fisher Information Matrix acts as a local map of the model's sensitivity. If you nudge a parameter and the model’s prediction changes drastically, that parameter has high Fisher information. If the prediction barely moves, the information is low.

This geometric perspective explains why some models generalize better than others. In A Geometric Modeling of Occam’s Razor in Deep Learning, Ke Sun and Frank Nielsen argue that the FIM provides a rigorous way to measure model complexity. Massive networks generalize well because their "effective" volume on this manifold is often much smaller than their parameter count suggests.

The Scalability Wall

If the FIM is so powerful, why don't we use it for every update? The problem is the Curse of Dimensionality. For a model with $N$ parameters, the FIM is an $N \times N$ matrix.

  • A 1-million parameter model requires a matrix with 1 trillion entries.
  • A 70-billion parameter LLM would require a matrix so large it would exceed the memory capacity of any existing supercomputer.

As noted by James Martens and Roger B. Grosse in their work on K-FAC, calculating and inverting this matrix scales quadratically and cubically, making exact second-order optimization "computationally prohibitive" for modern architectures.

Factoring the Curvature

To climb this wall, researchers have developed a suite of approximations that trade off precision for feasibility. The most influential of these is Kronecker-factored Approximate Curvature (K-FAC).

Instead of treating the entire network as one giant block, K-FAC approximates the FIM by assuming that the correlations between layers can be factored into smaller, manageable pieces. James Martens and Roger B. Grosse showed that for standard feed-forward and convolutional layers, the FIM can be viewed as a Kronecker product of two much smaller matrices. This reduces the storage and inversion cost from $O(N^2)$ to something closer to the cost of standard backpropagation.

More recent work has pushed this even further. Xiaolei Liu proposed Layer-Wise Natural Gradient (LNGD), which uses a diagonal-Kronecker hybrid to make second-order optimization practical for "frontier" models. Meanwhile, Ke Sun has explored the trade-offs of diagonal estimators, providing a theoretical guide for when simple diagonal approximations (like those used in Adam) are "good enough" versus when they fail to capture essential curvature.

Beyond Optimization: The Fisher as a Swiss Army Knife

The ability to approximate the FIM has opened doors far beyond just speeding up training. Because the Fisher matrix tells us which parameters "matter" for the model's current knowledge, it has become a cornerstone for several emerging fields:

  1. Model Merging: When combining two specialized models, simply averaging their weights often fails. Pouria Mahdavinia and colleagues developed OTA Merging, which uses discarded training statistics as a "free" map of curvature to merge models with surgical precision.
  2. Machine Unlearning: To satisfy the "right to be forgotten," models must erase specific data without retraining. Ryutaro Tanno in Repairing Neural Networks and Xiaoyu You in RRL use Fisher information to identify which weights were most influenced by the "culprit" data, allowing them to be adjusted without damaging the rest of the model's knowledge.
  3. Interpretability: Colin Raffel and Michael Matena introduced NPEFF, which factorizes per-example Fisher matrices to uncover the "computational strategies" (like how a model handles dates or sentiment) hidden within the parameter space.

The Path Forward

The research direction is clear: we are moving away from treating neural networks as black-box vectors and toward treating them as structured geometric objects. While the full Fisher Information Matrix remains an elusive ideal, the development of increasingly sophisticated approximations—from Kronecker factors to layer-wise norms—is turning information geometry from a theoretical curiosity into a practical toolkit for the next generation of massive AI systems.

Papers in this version

Showing 60 of 218 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 117 papers
  2. Version 1 · 2026-10-04 · 218 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_fisher_information_matrix_scalability_0b052e11,
  title = {Fisher Information Matrix scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fisher-information-matrix-scalability/rdr_0b052e11cbbf4963838266229434516b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scalable_fisher_information_for_deep_lea_0b052e11_v1,
  title = {Scalable Fisher Information for Deep Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-fisher-information-for-deep-learning/rdr_0b052e11cbbf4963838266229434516b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}