When optimizing a deep neural network, standard Stochastic Gradient Descent (SGD) takes a step in the direction of steepest descent within the parameter space. It assumes this space is flat and uniform. However, the relationship between a network's parameters and its actual predictions is highly non-linear. Tweaking a single weight in an early layer might completely alter the model's output distribution, while changing a weight in a later layer might have no noticeable effect.
This mismatch creates pathological curvature—steep ravines and flat plateaus—where first-order optimizers like SGD or Adam can oscillate or stall. To navigate this landscape efficiently, we need to measure distance not by how much the raw parameters change, but by how much the model's output distribution changes.
The Geometry of the Loss Landscape
In his foundational work on information geometry, Shun-ichi Amari proposed Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient, which redefines the steepest descent direction on a Riemannian manifold where distance is measured by the Kullback-Leibler (KL) divergence. The local metric tensor of this manifold is the Fisher Information Matrix (FIM), denoted as $F$.
The natural gradient update adjusts the standard gradient $\nabla L$ by multiplying it by the inverse of the Fisher matrix:
$$\Delta \theta = - \eta F^{-1} \nabla L$$
By conditioning the gradient with $F^{-1}$, Natural Gradient Descent (NGD) takes steps of constant size in the space of probability distributions, automatically accelerating through flat regions and slowing down near sharp curves.
The catch is scalability. For a model with $D$ parameters, the FIM is a $D \times D$ matrix. Storing it requires $O(D^2)$ memory, and computing its inverse requires $O(D^3)$ time. For modern neural networks with millions or billions of parameters, calculating $F^{-1}$ directly is computationally impossible. The field of second-order optimization is largely defined by finding structured approximations that make this matrix tractable without losing its geometric benefits.
Kronecker Factorization
The most successful breakthrough in scaling the Fisher matrix relies on exploiting the structural mechanics of backpropagation. Consider a single linear layer in a neural network. The input vector is $x$, the weight matrix is $W$, and the incoming gradient of the loss with respect to the pre-activation outputs is $g$. By the chain rule, the gradient with respect to the weight matrix is an outer product:
$$\nabla_W L = g x^T$$
The block of the Fisher matrix corresponding to this layer is the expected value of the outer product of these flattened gradients:
$$F_{\text{layer}} = \mathbb{E} \left[ \text{vec}(\nabla_W L) \text{vec}(\nabla_W L)^T \right] = \mathbb{E} \left[ (x x^T) \otimes (g g^T) \right]$$
where $\otimes$ denotes the Kronecker product.
To make this invertible, Kronecker-factored Approximate Curvature (K-FAC) introduces a crucial structural assumption: the covariance of the inputs $x$ and the covariance of the derivatives $g$ are statistically independent. This allows the expectation of the Kronecker product to be factored into the Kronecker product of two much smaller expectations:
$$F_{\text{layer}} \approx \mathbb{E}[x x^T] \otimes \mathbb{E}[g g^T] = A \otimes G$$
This factorization changes everything because of a clean algebraic property of Kronecker products: the inverse of a Kronecker product is the Kronecker product of the individual inverses:
$$(A \otimes G)^{-1} = A^{-1} \otimes G^{-1}$$
Instead of inverting a massive matrix of size $(N_{\text{in}} N_{\text{out}}) \times (N_{\text{in}} N_{\text{out}})$, we only need to invert two tiny matrices of size $N_{\text{in}} \times N_{\text{in}}$ and $N_{\text{out}} \times N_{\text{out}}$.
Grosse et al. extended this framework to convolutional networks in A Kronecker-factored approximate Fisher matrix for convolution layers by accounting for spatial weight sharing. To handle sequential dependencies, Martens et al. adapted the approach for recurrent architectures in Kronecker-factored Curvature Approximations for Recurrent Neural Networks. In distributed systems, Pauloski et al. demonstrated in Convolutional Neural Network Training with Distributed K-FAC that distributed eigen-decompositions can bypass explicit matrix inversions entirely, outperforming highly tuned first-order baselines.
Refining the Curvature Approximation
While K-FAC assumes strict independence between inputs and gradients, subsequent work has sought to correct the approximation errors this assumption introduces. George et al. proposed Eigenvalue-corrected Kronecker Factorization (EKFAC) in Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis. EKFAC computes the same Kronecker factors $A$ and $G$ to establish a shared coordinate system (an eigenbasis), but tracks the exact diagonal variances within that basis, correcting for the scaling inaccuracies that standard K-FAC overlooks.
Full Fisher Matrix [D x D] (Intractable)
│
▼
Block-Diagonal Approximation (Layer-by-Layer Independence)
│
▼
Kronecker Factorization (K-FAC) ──► Eigenbasis Correction (EKFAC)
[N_in x N_in] ⊗ [N_out x N_out] Rescales exact diagonal variances
Other approaches simplify the matrix by imposing block-diagonal constraints directly. Yun et al. developed AdaBlock: SGD with Practical Block Diagonal Matrix Adaptation for Deep Learning to bridge the gap between coordinate-wise adaptive methods like Adam and full-matrix second-order methods. Interestingly, this structural simplification mirrors the inherent physics of deep networks. In Fisher Information and Natural Gradient Learning in Random Deep Networks, Amari et al. mathematically proved that as random deep networks grow wider, the true Fisher Information Matrix naturally converges toward a unit-wise block diagonal structure, validating these localized approximation strategies.
Changing the Coordinate Space
Instead of spending compute to approximate and invert $F$, an alternative strategy is to reparameterize the network itself so that its natural coordinate system is already well-conditioned. If the parameter space can be transformed such that the Fisher matrix becomes the identity matrix ($F = I$), standard gradient descent will automatically behave like natural gradient descent.
The authors of Natural Neural Networks achieved this by introducing periodic "whitening" transformations directly into the network's internal representations. By decorrelating the activations and centering the gradients layer-by-layer, standard SGD updates effectively execute natural gradient steps.
This concept is also highly effective for preserving knowledge across tasks. In continual learning, Liu et al. demonstrated in Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting that rotating a network's parameter space to align with the principal components of the data makes the Fisher Information Matrix approximately diagonal. This allows simple diagonal consolidation algorithms to protect critical parameters without needing a computationally expensive full covariance matrix.
The Pitfalls of the Empirical Fisher
When implementing these approximations, it is tempting to use a common shortcut known as the empirical Fisher. The true Fisher matrix requires computing the expectation over the model's own predictive distribution $p_\theta(y|x)$, which means evaluating gradients on synthetic labels sampled from the model's output. The empirical Fisher replaces this by evaluating gradients directly on the true training labels.
While computationally convenient, this shortcut can fail catastrophically. In Limitations of the empirical Fisher approximation for natural gradient descent, Kunstner et al. demonstrated that the empirical Fisher does not preserve true second-order curvature when the model is far from the optimum. Instead, it distorts the optimization path by scaling updates based on the magnitude of the data gradients rather than the underlying shape of the loss landscape. To mitigate this distortion, Zhang et al. introduced an inversely scaled projection fix in An Improved Empirical Fisher Approximation for Natural Gradient Descent, restoring geometric accuracy while preserving the practical efficiency of empirical sampling.
Applications Beyond Optimization
Making the Fisher Information Matrix scalable does more than just accelerate training; it unlocks a suite of diagnostic and operational capabilities across machine learning.
- Uncertainty Quantification: Traditional frequentist variance propagation requires inverting the Fisher matrix. By bypassing this bottleneck, Kallus et al. developed The Implicit Delta Method to compute frequentist confidence intervals for high-dimensional models at a fraction of the traditional computational cost.
- Active Learning: Selecting the most informative data points to label often relies on variance reduction metrics. Ash introduced BAIT in Gone Fishing: Neural Active Learning with Fisher Embeddings, utilizing tractable Fisher matrix projections to select optimal batches of unlabeled data for training deep networks.
- Machine Unlearning: To satisfy privacy regulations without retraining a model from scratch, You et al. developed RRL: Recommendation Reverse Learning. This framework uses localized Fisher approximations to selectively erase the influence of specific user interactions while preserving the broader collaborative relationships embedded within the network weights.
- Out-of-Distribution Detection: Dauncey demonstrated in Approximations to the Fisher Information Metric of Deep Generative Models for Out-Of-Distribution Detection that layer-wise gradient norms can serve as a robust, representation-invariant proxy for the Fisher metric, successfully identifying anomalous data where standard likelihood tests fail.