Closed-Form Divergences and Tractable Information Geometry

This direction is split into other directions: analytical intractability of mixture entropy. Its last version stays readable and citable here.

Version 1 of 1131 papers142 conceptsfrozen 2026-10-04

In machine learning, we often treat probability distributions as if they were simple points in space. We want to measure the distance between them, find their average, or calculate their "volume" (entropy). However, for most distributions, these operations are mathematically impossible to perform exactly. We are often hitting a "closed-form wall": while we can write down the formula for a distribution, we cannot write down a simple formula for the distance between two of them.

This research direction focuses on breaking through that wall, developing the mathematical tools to make complex distributions—like mixtures, heavy-tailed Cauchy distributions, and directional models—as computationally efficient as the ubiquitous Gaussian.

The Closed-Form Wall

The reason the Gaussian distribution dominates machine learning is not just its natural occurrence, but its mathematical "politeness." If you take the Kullback-Leibler (KL) divergence between two Gaussians, the result is a clean, closed-form algebraic expression. You can plug it into a loss function and calculate gradients instantly.

But as soon as we move toward more realistic models, the math breaks. As Frank Nielsen points out in On a generalization of the Jensen-Shannon divergence, many parametric families are not "closed" under basic operations. For example, the arithmetic mixture of two Gaussians is no longer a Gaussian—it is a Gaussian Mixture Model (GMM).

This lack of closure creates a cascade of intractability:

  1. Entropy of Mixtures: There is no closed-form formula for the Shannon entropy of a GMM, as noted by Richard Nock and Frank Nielsen in A series of maximum entropy upper bounds.
  2. Divergence between Mixtures: Measuring the KL divergence between two GMMs requires expensive Monte Carlo sampling or numerical integration, which Jacob Goldberger and Sam T. Roweis highlight as a primary bottleneck for model compression in Hierarchical Clustering of a Mixture Model.
  3. Heavy Tails: For distributions like the Cauchy or alpha-stable families, standard metrics often result in infinite or intractable values, making them difficult to use in robust signal processing.

Generalizing Means to Restore Tractability

The core intuition behind much of this work is that the "intractability" of a divergence often stems from the choice of the mean used to compare distributions. Standard divergences like Jensen-Shannon use the arithmetic mean of two distributions.

In Constraining Variational Inference with Geometric Jensen-Shannon Divergence, Jacob Deasy and colleagues demonstrate that by switching from an arithmetic mean to a geometric mean, the divergence between exponential family distributions suddenly becomes closed-form. This "Skew-Geometric" approach allows Variational Autoencoders to balance different latent space behaviors without the computational cost of approximations.

Frank Nielsen generalizes this further in The α-divergences associated with a pair of strictly comparable quasi-arithmetic means. By picking the "right" abstract mean for a specific family of distributions, we can derive analytical formulas for distances that were previously thought to be purely numerical problems.

Analytical Breakthroughs for Non-Standard Models

Beyond generalizing means, researchers are finding specific mathematical "keys" to unlock difficult distributions:

Why It Matters: Beyond the Gaussian

The goal of this research is to allow machine learning models to be both expressive and efficient.

If we can calculate the distance between complex mixtures in closed form, we can perform real-time clustering of directional data on hyperspheres, as shown by Stephen Maybank in Generalised Watson Distribution on the Hypersphere. If we can calculate the entropy of a mixture, we can better regularize deep Bayesian networks or improve word similarity measures in NLP, as explored by Jun'ichi Kazama in A Bayesian Method for Robust Estimation of Distributional Similarities.

By expanding the library of tractable statistical tools, this field enables a move away from "Gaussian-only" modeling toward a more flexible and robust information geometry.

Papers in this version

Showing 60 of 131 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_closed_form_divergences_and_tractable_in_a4c3879a,
  title = {Closed-Form Divergences and Tractable Information Geometry},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/closed-form-divergences-and-tractable-information-geometry/rdr_a4c3879ab14344e8a353251f4707bd4d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_closed_form_divergences_and_tractable_in_a4c3879a_v1,
  title = {Closed-Form Divergences and Tractable Information Geometry},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/closed-form-divergences-and-tractable-information-geometry/rdr_a4c3879ab14344e8a353251f4707bd4d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}