Admixture of Poisson MRFs: A Topic Model with Word Dependencies

David I. Inouye, Pradeep Ravikumar, Inderjit S. DhillonICML 2014 · OpenReview

Traditional topic models like Latent Dirichlet Allocation (LDA) treat documents as "bags of words," ignoring the complex dependencies between specific terms. This paper introduces the Admixture of Poisson Markov Random Fields (APM), a framework that models topics as networks of dependent words, providing more intuitive visualizations and a mathematical generalization of the classic LDA model.

The "Bag of Words" Problem

Most popular topic models rely on the multinomial distribution, which carries a major flaw: it assumes words are independent within a topic.

In reality, words are highly dependent. If a document contains the word "classification," "supervised" becomes much more likely. Conversely, if a machine learning paper mentions "kernels," the word "graphs" might become less likely if the paper focuses specifically on Support Vector Machines. Standard models cannot capture these nuances, nor do they easily account for the absence of words or varying document lengths. The authors aim to bridge this gap by creating a model that treats topics as graphical structures rather than simple frequency lists.

Introducing the APM Framework

The authors propose the Admixture of Poisson Markov Random Fields (APM), built on several core components:

  • Poisson Markov Random Fields (PMRFs): Unlike multinomials, PMRFs model multivariate count data and explicitly include a dependency parameter. This allows the model to capture how the presence of one word increases or decreases the probability of another.
  • The Admixture Framework: While a standard mixture model assumes a document belongs to exactly one topic, an admixture allows a document to be a combination of multiple topics. APM performs this mixing in the "canonical parameter space" of the Poisson distribution.
  • Mathematical Equivalence: A key theoretical contribution is the proof that the conditional distribution of LDA is equivalent to a PMRF where all word dependencies are zero. This establishes APM as a true generalization of LDA.
  • Optimization and Sparsity: The authors use a proximal gradient algorithm with $\ell_1$ regularization. The $\ell_1$ penalty ensures that the resulting word-dependency graphs are "sparse," meaning only the most significant relationships between words are preserved for interpretability.

Key Insights: Topics as Networks

By enforcing sparsity, the model automatically discovers "interesting subgraphs"—coherent semantic concepts that emerge from the data without manual labeling. Instead of viewing a topic as a flat list (e.g., "music, theater, play"), APM views it as a graph.

APM Topic Network Visualization

This network approach reveals "hub words" that connect different semantic clusters. Furthermore, APM can model negative correlations (inhibitory relationships). For example, if "history" and "war" appear, the model can learn that "science" is less likely to appear, refining document categorization.

Experimental Results

The authors evaluated APM using the Grolier Encyclopedia (for qualitative graph visualization) and the CMU 20 Newsgroup Dataset (for quantitative coherence testing).

  • Qualitative Superiority: APM produces highly intuitive topic structures. For instance, in a "Temperature" topic, the word "temperature" acts as a central hub connecting "heat," "gas," and "deg."
  • Quantitative Coherence: APM significantly outperformed LDA on the UMass Coherence metric (internal fit) when the number of topics was small. On the PMI metric (external semantic validity), APM performed comparably to LDA, suggesting it maintains semantic relevance while providing rich structural information.

Limitations and Future Work

While powerful, APM introduces new challenges:

  • Scalability: The dependency matrix scales quadratically with vocabulary size, making it difficult to apply to massive vocabularies without further optimization.
  • Computational Cost: The proximal optimization required over 5,000 iterations to converge, suggesting faster methods (like Proximal Newton) are needed for practical use.
  • Hyperparameter Sensitivity: Choosing the right regularization parameter ($\lambda$) is crucial. If set too high, the model reverts to an independent Poisson model, losing the benefit of the dependency graphs.

Ultimately, APM provides a compelling new lens for topic modeling, proving that moving beyond the "bag of words" assumption can yield richer, more interpretable representations of text.

Summary updated

Abstract

This paper introduces a new topic model based on an admixture of Poisson Markov Random Fields (APM), which can model dependencies between words as opposed to previous independent topic models such ...
BibTeX
@inproceedings{inouye2014admixture,
  author = {David I. Inouye and Pradeep Ravikumar and Inderjit S. Dhillon},
  title = {{Admixture of Poisson MRFs: A Topic Model with Word Dependencies}},
  booktitle = {International Conference on Machine Learning},
  year = {2014},
  url = {https://openreview.net/forum?id=HJENyoWdWr},
}
Figure