A note on the relations between mixture models, maximum-likelihood and entropic optimal transport

Titouan Vayer, Etienne LasallearXiv 2025 · DOI · arXiv

TL;DR

This pedagogical note demonstrates that Maximum Likelihood Estimation (MLE) for mixture models is mathematically equivalent to solving an Entropic Optimal Transport (EOT) problem. By reframing the problem, the authors show that the classic Expectation-Maximization (EM) algorithm is actually a specific form of block-coordinate descent on an optimal transport loss.

What Problem Does This Paper Address?

In machine learning and statistics, we often treat Maximum Likelihood Estimation (finding parameters that make observed data most probable) and Optimal Transport (finding the most efficient way to move "mass" from one distribution to another) as two distinct toolkits.

While researchers have previously noted connections between these fields, the derivations are often buried in complex proofs or specific applications like deconvolution. This paper addresses the need for a concise, pedagogical bridge between the two. It specifically tackles the "discrete mixture model" case—where we assume data comes from a set of $K$ different sub-populations (like clusters in a Gaussian Mixture Model)—and shows that the math governing these models is identical to the math of entropic optimal transport.

Methods

The authors use a purely theoretical approach to establish this link:

  1. Variational Reformulation: They use the Gibbs variational principle (also known as the dual formulation of KL divergence) to rewrite the "log-sum-exp" term found in the negative log-likelihood of a mixture model. This transformation turns a log-probability into a maximization problem over a probability simplex.
  2. Mapping to OT: By introducing a cost matrix $C$ (defined as the negative log-probability of a data point given a cluster), they show that the negative log-likelihood is equivalent to a semi-relaxed entropic OT problem. In this setup, one marginal is fixed to the empirical data distribution, while the other (the cluster weights) is optimized.
  3. Algorithmic Equivalence: To illustrate the practical side, they take the standard EM algorithm for Gaussian Mixture Models (GMMs) and show that its "E-step" and "M-step" correspond exactly to alternating minimization steps (Block Coordinate Descent) on the entropic OT objective function.

Datasets

Not discussed in the paper. As a theoretical note, the work relies on mathematical proofs and does not include empirical experiments or benchmarking on specific datasets.

Key Insights and Ideas

  • Likelihood as Geometry: The paper reinforces the idea that statistical likelihood has a geometric interpretation. Minimizing the negative log-likelihood is equivalent to finding a "transport plan" that moves probability mass from your model's components to your observed data points with minimum "cost."
  • The Role of Entropy: The "entropic" part of EOT corresponds to the soft-assignment of data points to clusters. Without entropic regularization, OT would result in "hard" assignments; with it, we get the "soft" assignments characteristic of the EM algorithm.
  • EM is BCD: A major "aha moment" is seeing the EM algorithm not just as a statistical heuristic, but as a principled optimization strategy (Block Coordinate Descent) on a well-defined geometric loss function.
  • Weights as Marginals: In the OT framework, the mixture weights ($\pi$) are revealed to be the second marginal of the transport plan. Optimizing the weights is equivalent to adjusting the supply constraints in a transport problem.

Key Results

  • Proposition 2.3: This is the core result. It formally states that for any mixture model, the negative log-likelihood is equal to the minimum of a semi-relaxed entropic OT problem.
  • GMM Mapping: The authors provide explicit formulas showing that the standard updates for GMM means ($\mu$), covariances ($\Sigma$), and weights ($\pi$) are exactly the solutions to the coordinate descent steps of the EOT problem.

Limitations and Future Work

  • Edge Cases: The authors deliberately ignore "edge cases" (like $0 \log 0$) to keep the presentation simple and pedagogical, noting that a fully rigorous treatment would require reasoning on the supports of matrices.
  • Scope: The note focuses primarily on discrete mixtures. While they briefly mention that these results can be generalized to infinite mixtures and more complex generative models, they do not provide the full derivations for those cases.
  • Computational Perspective: While the paper links MLE to "Computational OT," it doesn't explore whether OT-specific solvers (like the Sinkhorn algorithm) could offer performance benefits over standard EM for certain types of mixture models.

Summary updated

BibTeX
@misc{vayer2025note,
  author = {Titouan Vayer and Etienne Lasalle},
  title = {{A note on the relations between mixture models, maximum-likelihood and entropic optimal transport}},
  year = {2025},
  eprint = {2501.12005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.12005},
}
Figure