Correlated Topic Models

David M. Blei, John D. LaffertyNeurIPS 2005 · OpenReview

TL;DR

While Latent Dirichlet Allocation (LDA) revolutionized how we find "topics" in text, it assumes topics are independent—an unrealistic constraint for real-world data. Correlated Topic Models (CTM) fix this by using a logistic normal distribution to capture how topics like "genetics" and "disease" often travel together. This results in better predictive performance and allows for intuitive "topic graphs" that help users explore large datasets.

What Problem Does This Paper Address?

In the early 2000s, Latent Dirichlet Allocation (LDA) became the gold standard for unsupervised text analysis. LDA treats a document as a mixture of topics, but it has a major blind spot: it assumes the proportions of these topics are strictly independent.

In the real world, topics are rarely isolated. If you are reading a paper about "Neural Networks," you are much more likely to see "Optimization" than "Marine Biology." Because LDA uses the Dirichlet distribution to model topic proportions, it struggles to represent these natural correlations. This limitation makes LDA less accurate for prediction and less useful for visualizing the complex, interconnected relationships between different fields of study.

Methods: The Logistic Normal Distribution

The authors introduce the Correlated Topic Model (CTM) to bridge this gap. The core innovation is replacing the Dirichlet distribution with the logistic normal distribution.

  1. The Logistic Normal: This distribution starts with a multivariate normal random variable, which can easily model correlations via a covariance matrix ($\Sigma$). It then maps this variable onto the "simplex" (a vector of probabilities that sums to one) using a softmax-like transformation.
  2. The Challenge of Non-Conjugacy: In standard LDA, the Dirichlet distribution is "conjugate" to the multinomial distribution of words, making the underlying math relatively straightforward. The logistic normal, however, is not conjugate.
  3. Variational Inference: To handle this mathematical hurdle, the authors developed a mean-field variational inference algorithm. They used Jensen's inequality to bound the log probability and applied a clever Taylor expansion to handle the problematic "log-sum-exp" term (the log normalizer) that arises from the logistic normal distribution.

Datasets

The authors evaluated CTM using historical archives from the journal Science, provided by JSTOR:

  • Large-scale Visualization: 16,351 articles spanning 1990–1999 were used to build a comprehensive "topic graph."
  • Quantitative Benchmarking: A smaller subset of 1,452 articles from 1960 was used for 10-fold cross-validation to compare CTM against standard LDA.
  • Vocabulary: After pruning rare and overly common terms, they used a vocabulary of 5,612 unique words.

Key Insights and Ideas

Topic Graph from Science Articles
  • Topic Graphs: Because CTM learns a covariance matrix between topics, we can visualize the results as a network. Nodes represent topics (e.g., "Cell Biology" or "Geology"), and edges represent strong correlations. This allows researchers to navigate a corpus by following related ideas rather than just searching keywords.
  • Predictive Synergy: CTM is better at predicting unseen words in a document. If the model sees a few words related to "Brain," it uses the learned correlations to increase the probability of words related to "Memory" or "Cognition," even if those specific words haven't appeared yet.
  • Handling Complexity: CTM effectively "supports" more topics than LDA. In their tests, LDA’s performance peaked at around 30 topics before overfitting, while CTM continued to improve up to 90 topics, suggesting it captures finer-grained structure without losing the "big picture."

Key Results

Held-out Log Likelihood Comparison
  • Better Fit: CTM consistently achieved a higher held-out log-likelihood than LDA across various topic counts. As shown in the performance comparisons, CTM avoids the sharp drop-off in likelihood that LDA experiences when the number of topics increases.
  • Perplexity Reduction: In predictive tasks, CTM reduced perplexity (a measure of uncertainty) by nearly 200 words—roughly a 10% improvement over LDA—when only a small portion of a document was observed.
  • Qualitative Discovery: The model successfully grouped articles into intuitive clusters (e.g., connecting "Cognitive Science" with "Visual Neuroscience") that were not explicitly labeled in the raw text.

Limitations and Future Work

  • Computational Cost: The primary drawback is speed. Because the logistic normal lacks conjugacy, the inference step requires iterative optimization (coordinate ascent) for every document, making it significantly more computationally expensive than the original LDA.
  • Parameter Explosion: The covariance matrix grows at a rate of $K^2$ (where $K$ is the number of topics). For models with thousands of topics, the number of parameters to estimate could become a bottleneck.
  • Dynamic Modeling: While this paper focuses on static correlations, the authors hint that future work could explore how these correlations evolve over time (e.g., how the link between "Computer Science" and "Biology" has strengthened over decades).

Summary updated

Abstract

Topic models, such as latent Dirichlet allocation (LDA), can be useful tools for the statistical analysis of document collections and other discrete data. The LDA model assumes that the words of each document arise from a mixture of topics, each of which is a distribution over the vocabulary. A limitation of LDA is the inability to model topic correlation even though, for example, a document about genetics is more likely to also be about disease than x-ray astronomy. This limitation stems from the use of the Dirichlet distribution to model the variability among the topic proportions. In this paper we develop the correlated topic model (CTM), where the topic proportions exhibit correlation via the logistic normal distribution [1]. We derive a mean-field variational inference algorithm for approximate posterior inference in this model, which is complicated by the fact that the logistic normal is not conjugate to the multinomial. The CTM gives a better fit than LDA on a collection of OCRed articles from the journal Science. Furthermore, the CTM provides a natural way of visualizing and exploring this and other unstructured data sets.
BibTeX
@inproceedings{blei2005correlated,
  author = {David M. Blei and John D. Lafferty},
  title = {{Correlated Topic Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2005},
  url = {https://openreview.net/forum?id=Sy4NNvW_bB},
}
Figure