Analyses of Multi-collection Corpora via Compound Topic Modeling

Clint P. George, Wei Xia, George Michailidis

Introduction

Standard topic models, such as Latent Dirichlet Allocation (LDA), typically treat a corpus as a homogeneous "flat" bag of documents. However, real-world data is frequently structured into natural partitions—research papers grouped by publication year, news articles by source, or reviews by star rating. Analyzing these multi-collection corpora presents a dilemma: modeling the entire corpus as one ignores group-specific nuances, while modeling collections separately leads to "topic alignment" issues, where the same theme is assigned different indices across models.

To address this, the authors introduce Compound Latent Dirichlet Allocation (cLDA). This hierarchical model shares a global set of topics across all collections while allowing for collection-specific topic proportions, enabling seamless comparative analysis.

The cLDA Framework

The core innovation of cLDA is its compound hierarchy, which adds a collection-level layer between the global topics and individual documents.

Plate notation for the cLDA model, illustrating the hierarchy from global topics to collection-specific mixtures.

As shown in the graphical model above:

  • Global Level: A shared set of topics ($\beta$) is defined for the entire corpus.
  • Collection Level: Each sub-collection $j$ possesses its own latent mixture of topics ($\pi_j$).
  • Document Level: Each document $d$ within collection $j$ draws its specific topic mixture ($\theta_{jd}$) using the collection-level mixture ($\pi_j$) as a base.

To perform inference, the authors developed an Augmented Gibbs Sampler (AGS). By utilizing auxiliary variables (unsigned Stirling numbers of the first kind), the AGS effectively manages the non-conjugate relationship between the collection and document layers, outperforming more complex methods like Metropolis Adjusted Langevin Monte Carlo (MMALA) in convergence speed.

Experimental Evaluation

The model was evaluated on several diverse datasets to test its robustness and interpretability:

  • NIPS 00-18: 2,741 research papers (1988–2005) partitioned into four time spans.
  • Yelp Reviews: 24,310 reviews partitioned by star ratings (1–5).
  • 20Newsgroups: 10,764 articles partitioned into four major subject groups.

Key Results and Interpretability

cLDA demonstrates superior predictive power, consistently achieving lower perplexity than standard LDA, particularly when sub-collections are highly distinct. Beyond performance metrics, the model provides high interpretability by allowing researchers to track the "signature" of specific topics across collections.

Topic probability trends in the NIPS dataset across four time periods, showing the evolution of research interests.

In the NIPS dataset (visualized above), cLDA automatically discovers temporal trends. For instance, the probability of "neural networks" significantly decreased from the late 80s to the mid-2000s, while "mcmc" (Markov Chain Monte Carlo) and "generative models" saw a steady rise in popularity. This alignment is possible because the topics are shared across all time periods.

Similarly, in the Yelp study, the model successfully mapped topics to sentiment. Topics related to "waiting time" and "bad service" showed a clear decrease in probability as star ratings increased, while "ambiance" and "food quality" topics became more prevalent in higher-rated reviews.

Conclusion

By bridging the gap between global thematic structure and collection-specific variations, cLDA offers a powerful tool for comparative text analysis. While it requires a pre-specified number of topics and involves more computational complexity than flat LDA, its ability to provide aligned, coherent topics across structured data makes it a significant advancement for researchers analyzing evolving or partitioned corpora.

Summary updated

Abstract

As electronically stored data grow in daily life, obtaining novel and relevant information becomes challenging in text mining. Thus people have sought statistical methods based on term frequency, matrix algebra, or topic modeling for text mining. Popular topic models have centered on one single text collection, which is deficient for comparative text analyses. We consider a setting where one can partition the corpus into subcollections. Each subcollection shares a common set of topics, but there exists relative variation in topic proportions among collections. Including any prior knowledge about the corpus (e.g. organization structure), we propose the compound latent Dirichlet allocation (cLDA) model, improving on previous work, encouraging generalizability, and depending less on user-input parameters. To identify the parameters of interest in cLDA, we study Markov chain Monte Carlo (MCMC) and variational inference approaches extensively, and suggest an efficient MCMC method. We evaluate cLDA qualitatively and quantitatively using both synthetic and real-world corpora. The usability study on some real-world corpora illustrates the superiority of cLDA to explore the underlying topics automatically but also model their connections and variations across multiple collections.
BibTeX
@inproceedings{george2019analyses,
  author = {Clint P. George and Wei Xia and George Michailidis},
  title = {{Analyses of Multi-collection Corpora via Compound Topic Modeling}},
  booktitle = {LOD},
  year = {2019},
  url = {https://openreview.net/forum?id=oTDyJWQ6XOw},
}
Figure