Moment-Matching Conditions for Exponential Families with Conditioning or Hidden Data

Justin DomkearXiv 2020 · OpenReview

Abstract

Maximum likelihood learning with exponential families leads to moment-matching of the sufficient statistics, a classic result. This can be generalized to conditional exponential families and/or when there are hidden data. This document gives a first-principles explanation of these generalized moment-matching conditions, along with a self-contained derivation.

Introduction

Maximum Likelihood Estimation (MLE) is a cornerstone of statistical learning. For the class of exponential families, MLE possesses an elegant and intuitive property: at the optimum, the "moments" (expected values) of the model's features must exactly match the moments observed in the empirical data.

However, real-world applications often involve complexities such as latent variables (hidden data) or conditional modeling (e.g., Conditional Random Fields). In these scenarios, the standard moment-matching equations become more intricate. This paper provides a unified, first-principles derivation that simplifies these conditions into a single, intuitive framework.

The Core Property of Exponential Families

The elegance of exponential families is rooted in the properties of the log-partition function, $A(\theta)$. A fundamental result in statistics is that the gradient of the log-partition function with respect to the parameters $\theta$ is exactly the expected value of the sufficient statistics $T(x)$:

$$\nabla A(\theta) = \mathbb{E}_{p_\theta}[T(x)]$$

When maximizing the log-likelihood for a standard exponential family, setting the gradient of the objective to zero leads directly to the condition where the empirical average of the statistics matches the model's expectation:

$$\hat{\mathbb{E}}[T(x)] = \mathbb{E}_{p_\theta}[T(x)]$$

Generalizing to Conditional and Hidden Data

In many machine learning tasks, we encounter two specific variations:

  1. Conditioning: We model $p(y|x)$ rather than the joint distribution $p(x, y)$.
  2. Hidden Data: We incorporate latent variables $u$ that are never observed in the dataset.

Rather than re-deriving optimality conditions for every new model architecture, we can use a unified "Fill-in" Mnemonic to determine the correct moment-matching condition.

The "Fill-in" Mnemonic

To find the optimality condition for any exponential family variant, follow these steps:

  • Step 1: Place the data expectation on the left and the model expectation on the right.
  • Step 2: If variables are "missing" from the data (e.g., hidden variables $u$), use the model's conditional distribution to "fill them in" on the data side.
  • Step 3: If variables are not defined by the model's distribution (e.g., the input $x$ in a conditional model), use the empirical data to "fill them in" on the model side.
  • Step 4: At the MLE optimum, these two sides must be equal.

Key Results: Four Classes of Optimality

The paper derives specific conditions for four major scenarios based on this logic:

  1. Standard Exponential Family: The empirical moments match the model moments. $$\hat{\mathbb{E}}[T(x)] = \mathbb{E}_{p_\theta}[T(x)]$$

  2. Conditional Exponential Family: Since the model does not define a distribution over $x$, the data "fills in" $x$ for the model side. $$\hat{\mathbb{E}}_{x,y}[T(x,y)] = \hat{\mathbb{E}}_x \mathbb{E}_{p_\theta(y|x)}[T(x,y)]$$

  3. Exponential Family with Hidden Variables: Since the data lacks $u$, the model "fills in" $u$ for the data side using the conditional distribution $p(u|x)$. $$\hat{\mathbb{E}}_x \mathbb{E}_{p_\theta(u|x)}[T(x,u)] = \mathbb{E}_{p_\theta(x,u)}[T(x,u)]$$

  4. Conditional with Hidden Variables: A combination of both strategies where data provides $x$ and the model provides $u$.

Limitations and Practical Considerations

While these derivations provide a clear theoretical target, the author notes that this framework is pedagogical rather than algorithmic. In practice, calculating these expectations is often NP-hard. Consequently, researchers must frequently turn to approximate inference methods—such as Variational Inference or Markov Chain Monte Carlo (MCMC)—to satisfy these moment-matching conditions when exact calculation is intractable.

By unifying these cases under a single mnemonic, this work transforms a series of complex algebraic derivations into a coherent and memorable statistical principle.

Summary updated

Abstract

Maximum likelihood learning with exponential families leads to moment-matching of the sufficient statistics, a classic result. This can be generalized to conditional exponential families and/or when there are hidden data. This document gives a first-principles explanation of these generalized moment-matching conditions, along with a self-contained derivation.
BibTeX
@misc{domke2020moment,
  author = {Justin Domke},
  title = {{Moment-Matching Conditions for Exponential Families with Conditioning or Hidden Data}},
  year = {2020},
  eprint = {2001.09771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.09771},
}

Loading related content...

Figure