On an equivalence between PLSI and LDA

Mark A. Girolami, Ata KabánSIGIR 2003 · DOI

TL;DR

This paper reveals that Probabilistic Latent Semantic Indexing (PLSI) is a specific instance of Latent Dirichlet Allocation (LDA) when LDA is fitted with a uniform prior. This equivalence resolves inconsistencies in PLSI's generative semantics and provides a principled way to understand its "heuristic" folding-in mechanisms within the more robust LDA framework.

What Problem Does This Paper Address?

Language modeling is crucial for information retrieval (IR), with approaches like Probabilistic Latent Semantic Indexing (PLSI) decomposing document-term probabilities using latent variables. While PLSI offers good performance, it suffers from inconsistent generative semantics, making it problematic for handling new documents. Latent Dirichlet Allocation (LDA) emerged as a more robust alternative with consistent generative semantics. However, the relationship between these two powerful models remained unclear, leaving questions about whether PLSI's perceived shortcomings could be addressed by understanding its connection to LDA.

Methods

The core of this paper is demonstrating the mathematical equivalence between PLSI and LDA. The authors show that PLSI can be viewed as a Maximum A Posteriori (MAP) estimation of an LDA model under a uniform Dirichlet prior. This involves:

  • LDA Formulation: Describing LDA as a generative process where documents are mixtures of topics, and topics are distributions over words.
  • MAP Estimation: Explaining how to estimate LDA parameters using a Maximum A Posteriori (MAP) approach, which finds the most probable parameter values given the observed data.
  • Equivalence Proof: Deriving the MAP estimation for LDA and showing that under a uniform Dirichlet prior (where all prior probabilities are equal), the resulting MAP estimates for the topic-document distributions are identical to the parameters learned by PLSI. This also extends to how queries are "folded-in" to the model.

Datasets

Not discussed in the paper. The paper focuses on theoretical equivalence and uses general corpus notation (D, V, w, d) rather than specific datasets.

Key Insights and Ideas

  • PLSI as a Special Case of LDA: The most significant insight is that PLSI isn't fundamentally different from LDA but rather a specific instantiation of it. This reframes PLSI's limitations not as inherent flaws but as consequences of its specific estimation setup.
  • Resolving PLSI's Generative Inconsistency: By showing PLSI is a MAP-LDA under a uniform prior, the paper explains why PLSI struggles with new documents. The LDA framework, with its consistent generative semantics, inherently handles this better.
  • Principled "Folding-In": The paper provides a theoretical justification for PLSI's "heuristic" query folding-in mechanism. Within the LDA equivalence, this folding-in becomes a principled MAP estimation of the Dirichlet variable for the query.
  • Understanding Perplexity Differences: The paper suggests that observed differences in perplexity between PLSI and LDA might stem from the approximate inference methods used, rather than fundamental model differences, especially when considering more advanced inference techniques like Expectation Propagation.

Key Results

The paper's primary result is the mathematical demonstration of equivalence:

  • PLSI is equivalent to a MAP/ML estimation of an LDA model with a uniform Dirichlet prior.
  • The parameters learned by PLSI correspond to the MAP estimates of the topic-document distributions in LDA under this specific prior.
  • The "heuristic" query folding-in in PLSI can be understood as a principled MAP estimation of the Dirichlet variable within the LDA framework.

The paper does not present quantitative results from experiments on specific datasets, as its focus is on theoretical derivation and conceptual clarification.

Limitations and Future Work

  • Approximate Inference: The paper acknowledges that exact inference for LDA is intractable, and approximate methods are used. It hints that the choice of approximate inference (e.g., variational methods vs. Expectation Propagation) can impact performance metrics like perplexity, suggesting that more accurate inference could further bridge the gap in practical performance.
  • Uniform Prior Assumption: The equivalence is specifically shown for a uniform Dirichlet prior. The paper doesn't extensively explore the implications of non-uniform priors on LDA and their relationship to PLSI variants.
  • Future Work: The authors imply that understanding this equivalence resolves issues around likelihood computation and query folding-in. Future work could involve exploring the practical benefits of fitting LDA with non-uniform priors and leveraging the principled estimation for improved IR performance.

Summary updated

BibTeX
@inproceedings{girolami2003equivalence,
  author = {Mark A. Girolami and Ata Kabán},
  title = {{On an equivalence between PLSI and LDA}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2003},
  doi = {10.1145/860435.860537},
}
Figure