Modeling posterior probabilities using the linear exponential family

Peder Olsen, Vaibhava Goel, Charles Micchelli, John R. HersheyInterspeech 2010

TL;DR

This paper introduces the "linear exponential family" as a numerically stable and effective alternative to the Dirichlet distribution for modeling posterior probabilities, particularly for use as features in automatic speech recognition. They present a novel method for computing its partition function, overcoming numerical stability issues, and demonstrate a tangible improvement in speech recognition accuracy.

What Problem Does This Paper Address?

In many machine learning tasks, especially in speech recognition, we often deal with probability distributions over discrete events. A common way to represent these probabilities is using the Dirichlet distribution, which is well-suited as a Bayesian prior. However, when we want to use these posterior probabilities as features within a larger model, like a Gaussian Mixture Model (GMM) for acoustic modeling, the Dirichlet distribution can present challenges. Specifically, its behavior near the boundaries of the probability simplex (where probabilities are close to 0 or 1) can lead to numerical instability and potential overtraining. The authors propose the linear exponential family as a more robust alternative for this specific application.

Methods

The core contribution of this paper lies in two main areas:

  • The Linear Exponential Family: They define and analyze the linear exponential family, which is a specific instance of a more general exponential family where the feature function is simply the identity. The key challenge they address is the computation of its partition function, $Z(\lambda)$, which is crucial for defining the probability distribution.
  • Numerically Stable Partition Function Computation: The direct computation of $Z(\lambda)$ for the linear exponential family can suffer from numerical instability, especially when parameters are close to each other. The authors leverage the concept of "divided differences" and Opitz's formula, which relates the partition function to the exponential of a matrix. This matrix exponential approach provides a numerically stable way to compute $Z(\lambda)$ and its gradient, even in challenging cases. They also discuss properties like invariance to permutation and shift invariance, which are important for practical use.

Datasets

  • Training Data: 50 hours of English Broadcast News Speech corpora (LDC97S44 and LDC98S71).
  • Testing Data: EARS Dev-04f set (3 hours of audio from 6 shows from November 2003).

Key Insights and Ideas

  • Linear Exponential Family as a Superior Feature: The paper argues that the linear exponential family offers desirable properties for posterior probability features, such as monotonicity and better behavior near simplex boundaries, compared to the Dirichlet distribution. This can lead to more robust models and reduce overtraining.
  • Matrix Exponential for Stability: The key to overcoming numerical issues in computing the partition function is to reframe the problem as computing the exponential of a specific matrix. This is a well-studied problem with robust numerical solutions.
  • Simultaneous Gradient Computation: The proposed matrix-based method allows for the simultaneous computation of the partition function and its gradient, which is essential for optimizing model parameters using gradient-based methods.

Key Results

  • Speech Recognition Improvement: When incorporating posterior probabilities modeled by the linear exponential family as additional features into an existing GMM-based acoustic model, the system achieved a 0.6% absolute reduction in word error rate (WER) on the Broadcast News LVCSR task.
  • Dirichlet Family Performance: The Dirichlet family, when used in the same experimental setup, did not yield an error rate reduction, suggesting potential overtraining issues.
  • Numerical Stability Demonstrated: The paper provides empirical evidence (Table 1 and Figure 3) showing that the direct computation of the partition function breaks down for larger $n$ or small $\epsilon$, while the Opitz's formula (matrix exponential) approach remains stable.

Limitations and Future Work

  • Overtraining Investigation: The authors are investigating whether the lack of improvement with the Dirichlet family is due to overtraining and whether the observed gains with the linear exponential family generalize to other test sets.
  • Computational Complexity: While the matrix exponential method is numerically stable, its computational complexity is $O(n^3 \log_2(n + ||\lambda||))$, which is higher than the direct computation's $O(n^2)$. However, the stability gains are presented as a worthwhile trade-off.
  • Further Exploration of Exponential Families: The paper suggests that the broader class of exponential families on the simplex could be further explored for similar applications.

Summary updated

BibTeX
@inproceedings{olsen2010modeling,
  author = {Peder Olsen and Vaibhava Goel and Charles Micchelli and John R. Hershey},
  title = {{Modeling posterior probabilities using the linear exponential family}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure