Efficient Natural Gradients via Dual Parameter Spaces

This direction is merged into another direction: natural gradient optimization. Its last version stays readable and citable here.

Version 1 of 116 papers21 conceptsfrozen 2026-10-04

Standard gradient descent assumes that the space of model parameters is "flat"—that a step of size $\epsilon$ in any direction has the same impact. However, in probabilistic modeling, this is rarely true. Changing a mean parameter by 0.1 might drastically shift a narrow Gaussian distribution while barely affecting a wide one.

To solve this, Natural Gradient Descent (NGD) scales updates by the Fisher Information Matrix (FIM), ensuring that steps are measured by how much they change the output distribution rather than the raw parameter values. While theoretically elegant, NGD is notoriously expensive because it requires inverting a massive FIM.

A growing body of research demonstrates that for many important models, we can bypass this inversion entirely. By leveraging the information geometry of exponential families, researchers have found that the "natural" update is often just a standard gradient update performed in a different coordinate system.

The Geometry of Probability

When we optimize a model, we are navigating a manifold of probability distributions. Youhei Akimoto and colleagues established a rigorous bridge between black-box optimization and information geometry, showing that successful algorithms like CMA-ES are essentially performing natural gradient ascent on this manifold Theoretical foundation for CMA-ES from information geometric perspective.

The core intuition is parameterization invariance. An ideal optimizer should reach the same solution regardless of whether you represent a distribution using its variance or its precision. Standard SGD lacks this property, but NGD achieves it by treating the FIM as a Riemannian metric that corrects for the "curvature" of the parameter space.

The Duality Shortcut

The most significant breakthrough in this field is the realization that for the exponential family of distributions (which includes Gaussians, Categoricals, and Poissons), there is a fundamental duality between two sets of parameters:

  1. Natural Parameters ($\theta$): These define the log-likelihood (e.g., $\mu/\sigma^2$ in a Gaussian).
  2. Expectation Parameters ($\eta$): These define the moments or expected values (e.g., the mean $\mu$).

Mohammad Emtiyaz Khan has shown that the natural gradient with respect to one set of parameters is simply the standard gradient with respect to the other Information Geometry of Variational Bayes. This "Bayesian Learning Rule" implies that we can achieve second-order optimization benefits using only first-order costs.

Instead of calculating $F^{-1}\nabla_\theta \mathcal{L}$, we simply compute $\nabla_\eta \mathcal{L}$. This dual-space update automatically accounts for the geometry of the distribution. Amber Hu and Scott Linderman applied this to state-space models in their SING framework, alternating between these representations to achieve 100x speedups in inference without ever explicitly inverting a matrix SING: SDE Inference via Natural Gradients.

Flattening the Landscape via Reparameterization

If the natural gradient is difficult to compute because the FIM is complex, why not change the model's structure so the FIM becomes simple?

Alessandro Manenti and Cesare Alippi proposed this for categorical variables. By using a hierarchical parameterization called catnat, they "flatten" the optimization landscape such that the FIM becomes diagonal Beyond Softmax: A Natural Parameterization for Categorical Random Variables. In this new coordinate system, standard gradient descent behaves exactly like natural gradient descent.

Similarly, in deep learning, Behnam Neyshabur and co-authors demonstrated that certain data-dependent normalizations are mathematically equivalent to a diagonal approximation of the natural gradient Data-Dependent Path Normalization in Neural Networks Behnam Neyshabur. By normalizing the "paths" through a network, they make the optimization invariant to weight rescaling, effectively pre-conditioning the problem.

Implicit Optimization in Complex Models

The power of these geometric insights extends to models where the partition function is usually intractable. Aurélien Decelle and Cyril Furtlehner showed that for Restricted Boltzmann Machines, natural gradient ascent can be framed as a constrained linear regression, bypassing the need for the noisy sampling typically required to estimate gradients Exact training of Restricted Boltzmann machines on intrinsically low dimensional data.

For more complex approximations, such as mixtures of distributions, Wu Lin and Mohammad Emtiyaz Khan developed the Minimal Conditional-Exponential Family (MCEF) framework. This allows natural gradient updates to be computed as simple gradients of expectation parameters even for multimodal distributions, which were previously considered too computationally heavy for NGD Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations.

By shifting the focus from matrix inversion to parameter representation, this research direction transforms the natural gradient from a theoretical ideal into a practical, scalable tool for high-dimensional optimization. Ivor W. Tsang summarizes this shift in the INGO framework: by operating directly in the natural parameter space, we remove the noise of matrix estimation and unlock state-of-the-art precision in non-smooth black-box tasks Stochastic Implicit Natural Gradient for Black-box Optimization.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_natural_gradients_via_dual_par_d8de33e0,
  title = {Efficient Natural Gradients via Dual Parameter Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-natural-gradients-via-dual-parameter-spaces/rdr_d8de33e088a24658bae5d3154ef0a3d6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_natural_gradients_via_dual_par_d8de33e0_v1,
  title = {Efficient Natural Gradients via Dual Parameter Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-natural-gradients-via-dual-parameter-spaces/rdr_d8de33e088a24658bae5d3154ef0a3d6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}