Introduction
Standard topic models, such as Latent Dirichlet Allocation (LDA), typically treat a corpus as a homogeneous "flat" bag of documents. However, real-world data is frequently structured into natural partitions—research papers grouped by publication year, news articles by source, or reviews by star rating. Analyzing these multi-collection corpora presents a dilemma: modeling the entire corpus as one ignores group-specific nuances, while modeling collections separately leads to "topic alignment" issues, where the same theme is assigned different indices across models.
To address this, the authors introduce Compound Latent Dirichlet Allocation (cLDA). This hierarchical model shares a global set of topics across all collections while allowing for collection-specific topic proportions, enabling seamless comparative analysis.
The cLDA Framework
The core innovation of cLDA is its compound hierarchy, which adds a collection-level layer between the global topics and individual documents.
As shown in the graphical model above:
- Global Level: A shared set of topics ($\beta$) is defined for the entire corpus.
- Collection Level: Each sub-collection $j$ possesses its own latent mixture of topics ($\pi_j$).
- Document Level: Each document $d$ within collection $j$ draws its specific topic mixture ($\theta_{jd}$) using the collection-level mixture ($\pi_j$) as a base.
To perform inference, the authors developed an Augmented Gibbs Sampler (AGS). By utilizing auxiliary variables (unsigned Stirling numbers of the first kind), the AGS effectively manages the non-conjugate relationship between the collection and document layers, outperforming more complex methods like Metropolis Adjusted Langevin Monte Carlo (MMALA) in convergence speed.
Experimental Evaluation
The model was evaluated on several diverse datasets to test its robustness and interpretability:
- NIPS 00-18: 2,741 research papers (1988–2005) partitioned into four time spans.
- Yelp Reviews: 24,310 reviews partitioned by star ratings (1–5).
- 20Newsgroups: 10,764 articles partitioned into four major subject groups.
Key Results and Interpretability
cLDA demonstrates superior predictive power, consistently achieving lower perplexity than standard LDA, particularly when sub-collections are highly distinct. Beyond performance metrics, the model provides high interpretability by allowing researchers to track the "signature" of specific topics across collections.
In the NIPS dataset (visualized above), cLDA automatically discovers temporal trends. For instance, the probability of "neural networks" significantly decreased from the late 80s to the mid-2000s, while "mcmc" (Markov Chain Monte Carlo) and "generative models" saw a steady rise in popularity. This alignment is possible because the topics are shared across all time periods.
Similarly, in the Yelp study, the model successfully mapped topics to sentiment. Topics related to "waiting time" and "bad service" showed a clear decrease in probability as star ratings increased, while "ambiance" and "food quality" topics became more prevalent in higher-rated reviews.
Conclusion
By bridging the gap between global thematic structure and collection-specific variations, cLDA offers a powerful tool for comparative text analysis. While it requires a pre-specified number of topics and involves more computational complexity than flat LDA, its ability to provide aligned, coherent topics across structured data makes it a significant advancement for researchers analyzing evolving or partitioned corpora.
Summary updated