Abstract
We develop Markov topic models (MTMs), a novel family of generative graphical models that can learn topics simultaneously from multiple corpora, such as papers from different conferences. We apply MTMs to discover shared themes and correlations across distinct document collections, providing a more nuanced understanding of how different communities relate to one another.
Introduction
Traditional topic modeling approaches, such as Latent Dirichlet Allocation (LDA), generally assume that a document collection is a single exchangeable set. However, real-world data is often naturally partitioned into distinct but related corpora, such as research papers from different academic conferences (e.g., ICML, NIPS, SIGIR). Treating these as a single bucket loses corpus-specific nuances, while modeling them independently ignores shared context.
Markov Topic Models (MTMs) bridge this gap. By linking local topic models through Gaussian Markov Random Fields (GMRFs), MTMs allow for the simultaneous discovery of topics across multiple corpora. This framework reveals not only the topics themselves but also the fine-grained correlations between different research communities on a theme-by-theme basis.
Methods
The MTM extends the standard LDA framework by introducing a Markovian structure to the topic parameters across different corpora.
Model Architecture
The model consists of two primary components:
- Local LDA Models: Each corpus is represented by its own local LDA structure, with specific document-topic distributions ($\theta$) and topic-word distributions ($\beta$).
- Gaussian Markov Random Fields (GMRFs): To capture dependencies, the natural parameters of the topic-word distributions across corpora are modeled as a joint Gaussian distribution. This allows the model to "tie" the same topic across different collections, facilitating information sharing.
A critical innovation in MTM is the use of a scaling factor ($g_{k,w}$) to ensure that high-probability words—those that define a topic—dominate the correlation estimation. This prevents the vast number of low-probability "noise" words from overwhelming the model's ability to find meaningful relationships.
Inference
Since the true posterior is intractable, the authors employ a variational Expectation-Maximization (EM) algorithm. Parameter updates are handled via numerical optimization (L-BFGS), and Iterative Proportional Fitting (IPF) is used to manage sparse GMRF structures efficiently.
Experimental Setup
The model was evaluated using a dataset of abstracts from six major computer science conferences held between 2005 and 2008: CIKM, ICML, KDD, NIPS, SIGIR, and WWW. The final dataset comprised 2,598 documents and approximately 170,000 words, with a vocabulary of 3,733 terms.
Results
MTMs were compared against standard LDA (pooling all data) and individual LDA models (trained per corpus).
Quantitative Performance
As shown in the experimental results, MTMs consistently achieved lower per-word predictive perplexity across all six conference datasets. This indicates that the MTM's ability to share information across corpora leads to a more accurate generative model of the text compared to baselines.
Notably, MTMs reached peak performance at a lower number of topics ($K \approx 10$) compared to standard LDA ($K \approx 20$). This suggests that the Markovian structure captures the underlying themes more efficiently than traditional methods.
Qualitative Insights
The model successfully mapped the "sociology" of computer science research. It revealed that while some conferences are highly correlated on theoretical topics (e.g., ICML and NIPS on "kernels"), they may be less related on application-focused themes. This high-resolution similarity mapping provides a more nuanced view than simple document clustering.
Discussion and Future Work
While MTMs offer significant advantages in modeling multi-corpus data, they introduce higher computational complexity due to the joint Gaussian modeling over large vocabularies. Future research could focus on automatically learning the GMRF structure—allowing the model to discover which corpora should be linked—rather than assuming a pre-defined relationship. Additionally, integrating MTMs with Dynamic Topic Models (DTM) could enable the tracking of cross-corpus correlations as they evolve over time.
Summary updated