Limitations of the empirical Fisher approximation for natural gradient descent

Frederik Kunstner, Philipp Hennig, Lukas Balles

TL;DR

This paper delivers a reality check to the machine learning community: the "empirical Fisher" (EF)—a popular, computationally cheap shortcut for the Fisher Information Matrix—is often a poor substitute for capturing curvature. The authors demonstrate that while the true Fisher captures second-order information (like the Hessian), the empirical version often fails to do so. This discrepancy can distort optimization paths and lead to pathological behavior far from the optimum.

What Problem Does This Paper Address?

In optimization, Natural Gradient Descent (NGD) is considered a gold standard for adapting to the geometry of a loss surface. It works by preconditioning gradient updates with the Fisher Information Matrix (FIM), which acts as a proxy for the Hessian (local curvature). However, calculating the true FIM is often computationally expensive.

To save time, many researchers and popular algorithms (like Adam or certain implementations of KFAC) use the empirical Fisher (EF). The EF is simply the sum of the outer products of the gradients of the individual data points. Because it uses the gradients you are already calculating, it is essentially "free" to compute.

The problem is that many practitioners assume the EF is a valid approximation of the FIM. This paper investigates whether that assumption holds water. Spoiler: it usually doesn't.

Methods

The authors use a combination of theoretical analysis and clear visualizations to dissect the differences between the Fisher, the Generalized Gauss-Newton (GGN) matrix, and the Empirical Fisher.

  1. Mathematical Comparison: They show that the true Fisher is an expectation over the model's predictive distribution, whereas the EF uses the actual training labels.
  2. Vector Field Visualization: They plot the gradient vector fields for standard Gradient Descent (GD), NGD, and EF-preconditioned descent on simple linear regression problems to see how the "preconditioner" actually changes the path to the minimum.
  3. Optimization Experiments: They compare the performance of these methods on several standard regression and classification datasets, specifically looking at how they behave far from the optimum.

Key Insights and Ideas

  • The Name is a Misnomer: The "empirical Fisher" is not actually a Monte Carlo estimate of the Fisher. A true Monte Carlo estimate would require sampling labels from the model's distribution; the EF uses the fixed training labels, which creates a fundamental theoretical gap.
  • The "Inverse Gradient" Scaling Problem: Far from the optimum, gradients are usually large. Since the EF is built from squared gradients, preconditioning with its inverse means that the larger the gradient, the smaller the step size becomes. This is often the exact opposite of what you want during early training.
  • Misspecification is Fatal: The EF only approaches the true Fisher at the minimum if the model is "correctly specified" (i.e., it can perfectly represent the data distribution). In real-world deep learning, models are almost always misspecified, meaning the EF can be arbitrarily wrong even when you've reached the bottom of the loss curve.
Quadratic approximations of the loss contour using the true Fisher versus the empirical Fisher under correct and misspecified model conditions.
  • Curvature vs. Noise: The authors suggest that the EF might be better understood as an adaptation to gradient noise (variance) in stochastic optimization, rather than an adaptation to curvature. This might explain why EF-based methods like Adam work in practice despite their flaws as curvature estimators.

Key Results

  • Distorted Gradients: Visualizations show that while NGD points directly toward the optimum by accounting for curvature, the EF often "twists" the gradient field, leading to inefficient, zig-zagging paths.
Comparison of optimization paths for Gradient Descent (GD), Natural Gradient Descent (NGD), and Empirical Fisher (EF) on a linear regression dataset.
  • Poor Directionality: In experiments on the Boston dataset, the cosine similarity between the EF-preconditioned update and the true natural gradient was often near zero or even negative, meaning the EF was sending the optimizer in the wrong direction.
  • Performance Gap: On almost all tested datasets (including a1a, BreastCancer, Boston Housing, Yacht Hydrodynamics, Powerplant, Wine, and Energy), NGD significantly outperformed EF-preconditioned descent in terms of loss reduction per iteration, especially in the early stages of training.

Limitations and Future Work

The authors acknowledge that their study focuses on the deterministic (full-batch) setting to isolate the effects of curvature. In the real world, most people use stochastic (mini-batch) optimization.

The paper concludes by suggesting that the practical success of EF-based methods isn't due to them being "approximate second-order methods." Instead, they likely succeed because they adapt to the variance of the gradients. The authors call for more research into this "variance adaptation" perspective to build a more solid theoretical foundation for the heuristics we use every day.

Summary updated

Abstract

Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-order methods and heuristics like Adam. We dispute this argument by showing that the empirical Fisher---unlike the Fisher---does not generally capture second-order information. We further argue that the conditions under which the empirical Fisher approaches the Fisher (and the Hessian) are unlikely to be met in practice, and that, even on simple optimization problems, the pathologies of the empirical Fisher can have undesirable effects.
BibTeX
@inproceedings{kunstner2019limitations,
  author = {Frederik Kunstner and Philipp Hennig and Lukas Balles},
  title = {{Limitations of the empirical Fisher approximation for natural gradient descent}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure