Maximum Likelihood

Paul Breiding, Kathlén Kohn, Bernd SturmfelsOberwolfach 2024 · DOI · OpenAlex

Abstract

In Chapter 2, we discussed the problem of minimizing the Euclidean distance from a data point $u$ to a model $X$ in $\mathbb{R}^{n}$ that is described by polynomial equations. In Chapter 5, we studied the analogous problem in the setting of algebraic statistics, where the model $X$ represents a family of probability distributions, and we used the Wasserstein metric to measure the distance from $u$ to $X$. Finally, in Chapter 9, we considered the distance from a polynomial system $u$ to the discriminant in the context of numerical analysis. This chapter reinterprets Maximum Likelihood Estimation (MLE) as a geometric optimization problem, replacing Euclidean distance with Kullback–Leibler (KL) divergence. By introducing the "ML degree"—the number of complex critical points for a given model—the authors provide a framework to measure the algebraic complexity of statistical inference across discrete, Gaussian, and even particle physics models.

Introduction

In classical statistics, Maximum Likelihood Estimation (MLE) is the standard for parameter estimation. However, the objective functions (log-likelihoods) often involve logarithms, making them non-algebraic. This research addresses this by showing that the partial derivatives of these functions are actually rational functions. This allows for the application of tools from Metric Algebraic Geometry to statistical models.

A simple discrete statistical model represented as a tree, where parameters (s) determine the probabilities of outcomes (p).

The central challenge is determining the "ML degree" of a model: given a variety $X$ (representing the statistical model) and a data vector $u$, how many potential solutions (critical points) exist? This is the information-theoretic analogue to the "ED degree" used for Euclidean distance problems.

Methodology

The authors employ several sophisticated algebraic and topological techniques to analyze these models:

  • Augmented Jacobians: To find the MLE, they construct an "augmented Jacobian matrix" by prepending the gradient of the log-likelihood function to the Jacobian of the model's defining equations. The ML degree is then the degree of the resulting critical ideal.
  • Topological Euler Characteristics: For "very affine varieties" (subvarieties of an algebraic torus), the ML degree is shown to be equal to the signed topological Euler characteristic of the manifold.
  • Duality and Hadamard Products: For determinantal varieties (like low-rank matrix models), they use a "multiplicative" version of duality, showing a bijection between critical points of dual models using the Hadamard (entrywise) product.
  • Numerical Algebraic Geometry: Practical implementation relies on software like HomotopyContinuation.jl to solve the resulting polynomial systems.

Key Insights and Results

MLE as Geometry

Minimizing KL divergence is geometrically equivalent to finding the "closest" point on a variety, but the "ruler" used is information-theoretic rather than Euclidean. Models with an ML degree of 1 are highly desirable because the MLE is a rational function of the data, meaning there is a closed-form formula for the best-fit parameters.

The Physics Connection

In a fascinating cross-disciplinary leap, the authors show that the "scattering equations" used in particle physics to describe amplitudes are mathematically identical to finding the critical points of a log-likelihood function for a specific geometric model (the CEGM model). For the $k=2$ scattering model, the ML degree is exactly $(m-3)!$, where $m$ is the number of particles.

Gaussian and Determinantal Models

For Gaussian models, the ML degree of a linear space of symmetric matrices exhibits a specific symmetry. For a $4 \times 4$ covariance matrix, the ML degree ranges from 3 to 21 depending on the dimension of the linear subspace. Furthermore, Theorem 11.12 provides a bijection between critical points of rank-$r$ models and rank-$(n_1 - r + 1)$ models, preserving reality and positivity.

Discussion and Future Work

While the theory provides robust bounds, several challenges remain. The ML degree counts solutions over the complex numbers, but in statistics, only real, positive solutions that sum to one are valid. Determining the number of real solutions remains a much harder problem. Additionally, much of the topological theory assumes the variety is nonsingular; extending these results to singular varieties requires advanced tools like Chern-Schwartz-MacPherson classes. Finally, the computation of thousands or millions of critical points for high-dimensional models requires massive parallel computing and specialized software.

Conclusion

By framing Maximum Likelihood Estimation within the context of algebraic geometry, this work provides a rigorous framework for understanding the complexity of statistical inference. The introduction of the ML degree offers a powerful metric for algebraic complexity that bridges the gap between information theory, geometry, and high-energy physics.

Summary updated

Abstract

Abstract In Chapter 2, we discussed the problem of minimizing the Euclidean distance from a data point u to a model X in $$\mathbb{R}^{n}$$ that is described by polynomial equations. In Chapter 5, we studied the analogous problem in the setting of algebraic statistics [167], where the model X represents a family of probability distributions, and we used the Wasserstein metric to measure the distance from u to X . Finally, in Chapter 9, we considered the distance from a polynomial system u to the discriminant in the context of numerical analysis.
BibTeX
@inproceedings{breiding2024maximum,
  author = {Paul Breiding and Kathlén Kohn and Bernd Sturmfels},
  title = {{Maximum Likelihood}},
  booktitle = {Oberwolfach},
  year = {2024},
  doi = {10.1007/978-3-031-51462-3\_11},
}
Figure