Integrating Document Clustering and Topic Modeling

Pengtao Xie, Eric P. Xing

Introduction

Document clustering and topic modeling are fundamental tasks in text mining. While clustering groups similar documents, topic modeling identifies latent themes. These tasks are inherently linked: better clusters lead to more coherent topics, and vice versa. However, traditional methods often treat them as separate steps—such as running LDA and then applying K-means to the output—which can lead to suboptimal results and semantic confusion.

This paper presents the Multi-Grain Clustering Topic Model (MGCTM), a unified generative framework that optimizes both tasks simultaneously. By coupling these processes, the model achieves superior clustering accuracy while extracting highly coherent "local" topics specific to individual clusters and "global" topics shared across the entire corpus.

The Multi-Grain Clustering Topic Model (MGCTM)

The core innovation of MGCTM is its ability to distinguish between "global" topics shared across the entire corpus and "local" topics specific to individual clusters. This prevents "topic bleed," where general terms or irrelevant cluster-specific terms contaminate other groups. For example, a "computer architecture" topic is prevented from appearing in an economics cluster because the model explicitly separates these semantic grains.

Generative Process

The model follows a structured generative process for each document:

  1. Cluster Assignment: A document is assigned to a latent cluster.
  2. Topic Selection: The document draws from a distribution of global topics (shared by all groups) and a distribution of local topics (exclusive to its assigned cluster).
  3. Word Generation: For each word, a Bernoulli variable (a coin flip) determines whether it is drawn from a global topic or a local topic.
Graphical representation of the MGCTM model architecture showing the relationship between latent variables and observed words.

To handle the complexity of this joint distribution, the authors utilize Variational Inference. This iterative approach approximates the posterior distribution of hidden variables, effectively optimizing clustering and topic extraction in tandem.

Experimental Evaluation

The MGCTM was evaluated on two standard benchmarks: the Reuters-21578 dataset and the 20-Newsgroups dataset. It was compared against standard clustering algorithms (K-means, Normalized Cut), matrix factorization (NMF, LSI), and pipeline approaches (LDA + K-means).

Clustering Performance

MGCTM demonstrated superior performance across all benchmarks. On the 20-Newsgroups dataset, it achieved an accuracy (AC) of 58.69%, significantly higher than the 33.65% achieved by standard K-means and the 45.63% of the competing joint model, the Cluster-based Topic Model (CTM).

Clustering accuracy (AC) comparison across different models and datasets, highlighting MGCTM's performance.

Topic Interpretability

Beyond quantitative metrics, human evaluation confirmed that MGCTM produces more coherent and relevant topics. By filtering background noise into global topics, the local topics become highly discriminative. In a blind study with five human annotators, MGCTM's topics were rated as more coherent (33.47% coherence measure) than those from CTM (31.60%). Annotators found that local topics were highly relevant to their assigned clusters, proving the model successfully isolates group-specific semantics.

Conclusion

The MGCTM framework successfully integrates document clustering and topic modeling, proving that joint optimization leads to mutual improvement in both tasks. By isolating local semantics from global noise, the model provides a clearer and more accurate representation of text corpora. Future research may explore semi-supervised extensions to incorporate prior knowledge, such as "must-link" constraints, into the clustering process.

Summary updated

BibTeX
@inproceedings{xie2013integrating,
  author = {Pengtao Xie and Eric P. Xing},
  title = {{Integrating Document Clustering and Topic Modeling}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2013},
  url = {https://openreview.net/forum?id=exP4GTU4AFO},
}
Figure