Toric invariant theory for maximum likelihood estimation in log-linear models

Carlos Améndola, Kathlén Kohn, Philipp Reichenbach, Anna SeigalAlgebraic 2021 · DOI

TL;DR

This paper bridges algebraic geometry and statistics by connecting toric invariant theory with maximum likelihood estimation (MLE) for log-linear models. The authors show that a fundamental problem in invariant theory – finding a point of minimal norm in an orbit under a torus action – is equivalent to finding the MLE. This connection provides new ways to understand and compute MLEs, particularly regarding their existence.

What Problem Does This Paper Address?

Maximum likelihood estimation (MLE) is a cornerstone of statistical inference, aiming to find the model parameters that best explain observed data. For log-linear models, a widely used class of models for categorical data, finding the MLE can be challenging, especially concerning its existence. While MLE is equivalent to minimizing the Kullback-Leibler (KL) divergence to the empirical distribution, the existence of a minimum is not always guaranteed. This paper tackles the problem of characterizing the existence of the MLE for log-linear models and providing computational insights by leveraging tools from algebraic geometry, specifically toric invariant theory.

Methods

The core of the paper lies in establishing a deep connection between two seemingly disparate fields:

  • Log-linear Models and MLE: The paper starts by defining log-linear models as probability distributions whose log-probabilities lie in a specific linear space. It frames MLE as finding a point in this model that minimizes the KL divergence to the observed data.
  • Toric Invariant Theory: The authors introduce concepts from invariant theory, particularly the action of a complex torus on a vector space. Key notions include "stability" (unstable, semistable, polystable, stable) of a vector under this action, which are characterized by polyhedral conditions related to a matrix defining the torus action. The Hilbert-Mumford criterion is central here.
  • Kempf-Ness Theorem: This theorem, crucial for connecting the two fields, relates points of minimal norm in an orbit closure to the vanishing of a moment map. The paper adapts this theorem for torus actions.
  • Moment Map: The moment map, a tool from geometric invariant theory, is used to link norm minimization to the structure of the polytope defined by the torus action.

The main contribution is showing that the existence and uniqueness of the MLE for a log-linear model are directly characterized by the stability properties (specifically, polystability) of a particular vector under a related torus action. Furthermore, the MLE itself can be computed from a point of minimal norm in an orbit, as identified by the vanishing of the moment map.

Datasets

Not discussed in the paper. The paper focuses on theoretical connections and general models rather than specific empirical datasets.

Key Insights and Ideas

  • Bridging Algebra and Statistics: The paper demonstrates a powerful synergy between algebraic geometry (toric invariant theory) and statistics (MLE for log-linear models). This suggests a broader framework for understanding statistical problems through geometric lenses.
  • Stability as an Existence Criterion: The notion of "stability" under a torus action, a concept from invariant theory, provides a precise geometric criterion for the existence and uniqueness of the MLE in log-linear models. Polystability, in particular, is shown to be equivalent to the existence of a unique MLE.
  • Norm Minimization as MLE Computation: The paper reveals that finding a point of minimal norm in an orbit under a torus action is equivalent to computing the MLE. This opens up avenues for using optimization algorithms from invariant theory to solve statistical estimation problems.
  • Generalization of Scaling Algorithms: The connection to norm minimization and orbit closures provides a unifying perspective on various scaling algorithms (like Sinkhorn scaling and Iterative Proportional Scaling) used in statistics and computer science.

Key Results

  • Theorem 4.3: Establishes the equivalence between the stability of a vector under a specific torus action and the existence/uniqueness of the MLE for a log-linear model. Specifically:
  • Semistable $\iff$ extended MLE exists and is unique.
  • Polystable $\iff$ MLE exists and is unique.
  • Theorem 4.7: Shows that the extended MLE can be obtained from a point $q$ in the orbit closure where the moment map vanishes, with its $j$-th entry given by $|q_j|^2 / ||q||^2$. If the vector is polystable, this point is the MLE.
  • Propositions 3.6 & 3.7: Connect the null cone (set of unstable points) to maximal and minimal sub-polytopes, providing geometric insights into the structure of unstable configurations.

Limitations and Future Work

  • Computational Complexity: While the paper establishes theoretical connections, the practical implementation of these invariant theory-based methods for computing MLEs might involve significant computational complexity, especially for high-dimensional problems. The paper mentions connections to convex optimization algorithms for norm minimization, which could be a path forward.
  • Generalization to Other Models: The paper focuses on log-linear models. A natural extension would be to explore whether similar connections exist for other classes of statistical models, particularly other exponential families. The authors explicitly pose this as a future research question in Section 6.3.
  • Comparison with Gaussian Models: The paper draws parallels and contrasts with multivariate Gaussian models, suggesting that the unifying concept linking invariant theory to MLE might extend to broader families of models.

Summary updated

Abstract

We establish connections between invariant theory and maximum likelihood estimation for discrete statistical models. We show that norm minimization over a torus orbit is equivalent to maximum likelihood estimation in log-linear models. We use notions of stability under a torus action to characterize the existence of the maximum likelihood estimate, and discuss connections to scaling algorithms.
BibTeX
@inproceedings{amendola2021toric,
  author = {Carlos Améndola and Kathlén Kohn and Philipp Reichenbach and Anna Seigal},
  title = {{Toric invariant theory for maximum likelihood estimation in log-linear models}},
  booktitle = {Algebraic},
  year = {2021},
  doi = {10.2140/astat.2021.12.187},
}
Figure