Analyses of Multi-collection Corpora via Compound Topic Modeling

Clint P. George, Wei Xia, George Michailidis

TL;DR

The paper introduces Compound Latent Dirichlet Allocation (cLDA), a hierarchical topic model designed to analyze corpora that are naturally partitioned into sub-collections (e.g., different years of a conference or different news sources). By sharing a global set of topics while allowing for collection-specific topic proportions, cLDA enables researchers to perform comparative text analysis without the "topic alignment" headaches of standard models.

What Problem Does This Paper Address?

Standard topic models like Latent Dirichlet Allocation (LDA) typically treat a whole corpus as a "flat" bag of documents. However, real-world data is often structured: news articles come from different outlets, research papers are published in different years, and reviews are associated with different star ratings.

If you want to compare how topics evolve or differ across these groups, you face two bad options with standard LDA:

  1. Model the whole corpus at once: This ignores the group structure, potentially washing out subtle variations between collections.
  2. Model each collection separately: This creates a "topic alignment" problem. Topic #1 in the 1990s collection might be Topic #5 in the 2000s collection, making direct comparison nearly impossible.

The authors propose cLDA to bridge this gap, providing a unified framework that captures both the shared thematic structure of the whole corpus and the unique "signature" of each sub-collection.

Methods

The core innovation is the compound hierarchy. In standard LDA, documents draw their topics from a single global distribution. In cLDA, the authors add a layer:

  • Global Level: A shared set of topics ($\beta$) exists for the entire corpus.
  • Collection Level: Each sub-collection $j$ has its own latent mixture of topics ($\pi_j$).
  • Document Level: Each document $d$ in collection $j$ draws its specific topic mixture ($\theta_{jd}$) using the collection-level mixture ($\pi_j$) as a base.

To make this model work, the authors explored several inference techniques to estimate these hidden parameters:

  • Augmented Gibbs Sampler (AGS): This is their recommended approach. It uses auxiliary variables (specifically unsigned Stirling numbers of the first kind) to handle the complex, non-conjugate relationship between the collection and document layers.
  • Metropolis Adjusted Langevin Monte Carlo (MMALA): A more modern approach that uses gradients to navigate the probability space, though the authors found it slower to converge than AGS in this specific setting.
  • Variational Expectation Maximization (VEM): A faster, deterministic approximation that the authors found to be less accurate than the MCMC methods.

Datasets

The authors evaluated cLDA on three diverse real-world datasets:

  1. NIPS 00-18: 2,741 research papers from the NIPS conference (1988–2005), partitioned into four time spans to track research trends.
  2. Yelp Reviews: 24,310 restaurant reviews, partitioned by customer ratings (1 to 5 stars) to see how topics correlate with sentiment.
  3. 20Newsgroups: 10,764 articles partitioned into four major subject groups (Computers, Recreation, Science, and Politics).
  4. Synthetic Data: Used for rigorous "ground truth" testing of their inference algorithms.

Key Insights and Ideas

  • Shared Vocabulary, Unique Dialects: By sharing topics across the entire corpus, cLDA ensures that "Topic 8" always means the same thing (e.g., "Neural Networks"), allowing you to see exactly how its popularity waxes and wanes across different years or ratings.
  • AGS Efficiency: Surprisingly, the "traditional" Augmented Gibbs Sampler outperformed the more mathematically complex Langevin dynamics (MMALA). It reached optimal parameter regions in significantly fewer iterations.
  • Automatic Trend Discovery: The model is unsupervised. You don't tell it what the topics are; you only provide the documents and their collection labels (like "Year: 1992"). The model then automatically discovers that "MCMC" became a hot topic in the late 90s while "ANN" (Artificial Neural Networks) dipped in popularity during the same period.

Key Results

  • Better Predictive Power: cLDA consistently achieved lower perplexity (a measure of how well the model predicts unseen data) than standard LDA. The gain was most significant when the sub-collections were highly distinct (like the different categories in 20Newsgroups).
  • Higher Coherence: Topics generated by cLDA were found to be more "coherent"—meaning the top words in a topic were more logically related to each other than those produced by LDA.
  • Interpretability: In the Yelp study, cLDA successfully mapped specific topics to star ratings. For example, topics related to "waiting time" and "bad service" showed a clear gradual decrease in probability as star ratings moved from 1 to 5, while "ambiance" and "food quality" topics increased.
  • Superiority over HDP: Compared to Hierarchical Dirichlet Processes (HDP), which automatically choose the number of topics, cLDA produced fewer "fragmented" or redundant topics, making it more useful for human experts.

Limitations and Future Work

  • Computational Overhead: While the authors claim the overhead is "low," the hierarchical nature of cLDA is inherently more complex than standard LDA, which might limit scaling on truly massive (billion-word) datasets without further optimization.
  • Fixed Topic Count: Like LDA, cLDA requires the user to pre-specify the number of topics ($K$). While they provide guidelines for choosing $K$, it still requires some trial and error.
  • VEM Performance: Their Variational Inference (VEM) implementation was found to be suboptimal. They suggest that future work could involve developing more efficient, parallelized versions of VEM to handle larger-scale data.

Summary updated

Abstract

Popular probabilistic topic models have typically centered on one single text collection, which is deficient for comparative text analyses. We consider a setting where we have partitionable corpora. Each subcollection shares a single set of topics, but there exists relative variation in topic proportions among collections. We propose the compound latent Dirichlet allocation (cLDA) model that encourages generalizability, depends less on user-input parameters, and includes any prior knowledge corpus organization structure. For parameter estimation, we study Markov chain Monte Carlo (MCMC) and variational inference approaches extensively and suggest an efficient MCMC method. We evaluate cLDA using both synthetic and real-world corpora and cLDA shows superior performance over the state-of-the-art models.
BibTeX
@inproceedings{george2019analyses,
  author = {Clint P. George and Wei Xia and George Michailidis},
  title = {{Analyses of Multi-collection Corpora via Compound Topic Modeling}},
  booktitle = {LOD},
  year = {2019},
  url = {https://openreview.net/forum?id=oTDyJWQ6XOw},
}

Loading related content...

Figure