Abstract
Topic models, such as Latent Dirichlet Allocation (LDA), are essential tools for the statistical analysis of large document collections. LDA assumes that documents are composed of a mixture of topics, where each topic is a distribution over a vocabulary. However, a significant limitation of LDA is its inability to model correlations between topics; for instance, a document discussing genetics is statistically more likely to discuss disease than X-ray astronomy. This limitation arises from the use of the Dirichlet distribution to model topic proportions. In this paper, we introduce the Correlated Topic Model (CTM), which utilizes the logistic normal distribution to capture topic correlations. We develop a fast variational inference algorithm to handle the non-conjugacy of the logistic normal and multinomial distributions. Applied to a decade of Science magazine articles (1990–1999), the CTM demonstrates a superior fit to the data compared to LDA and serves as a powerful exploratory tool for navigating large-scale scientific corpora.
Introduction: Beyond the Independence Assumption
In the landscape of automated document analysis, Latent Dirichlet Allocation (LDA) long served as the foundational framework for discovering latent themes in text. By treating documents as mixtures of topics, LDA allowed researchers to summarize millions of words into interpretable clusters. However, LDA operates under a restrictive assumption: the presence of one topic in a document is independent of the presence of any other.
In reality, ideas are deeply interconnected. Scientific discourse, in particular, is characterized by these relationships. A paper discussing "Neural Networks" is naturally more likely to involve "Optimization" or "Data Mining" than "Marine Biology." Because LDA relies on the Dirichlet distribution to model topic proportions, it lacks the mathematical machinery to represent these dependencies. The Correlated Topic Model (CTM) was developed to bridge this gap, providing a more realistic and nuanced map of information.
The Correlated Topic Model (CTM)
The core innovation of the CTM is the replacement of the Dirichlet distribution with the Logistic Normal distribution.
Modeling Correlation
Unlike the Dirichlet distribution, the logistic normal distribution allows for a full covariance matrix between topics. This structural shift enables the model to explicitly learn which topics tend to co-occur. By estimating this covariance, the CTM can identify that Topic A and Topic B are positively correlated, while Topic A and Topic C might be entirely unrelated. This results in a "topic graph"—a network of ideas where topics act as nodes and correlations act as edges.
Variational Inference and Mathematical Correction
Transitioning to the logistic normal distribution introduces significant computational challenges. In LDA, the Dirichlet distribution is "conjugate" to the multinomial distribution of words, allowing for relatively straightforward posterior inference. The logistic normal is not conjugate, making exact inference intractable.
To address this, the authors developed a fast variational inference algorithm. This approach approximates the complex posterior distribution, allowing the model to scale to massive datasets. It is worth noting that the original publication of this work required a formal correction regarding the variational lower bound equation—the objective function used for training. Specifically, an entropy term, $H(q_d)$, was omitted in the initial derivation, a detail critical for the mathematical integrity of the approximate inference.
Empirical Evaluation: A Decade of Science
To test the CTM's efficacy, the authors applied it to a substantial corpus: the Science magazine archives from 1990 to 1999. This dataset consists of approximately 57 million words across thousands of articles.
The goal was to determine if the CTM could better represent the diverse and overlapping fields of modern science. By analyzing this decade of research, the model was tasked with identifying clusters of related disciplines—such as linking various sub-fields of biology together while maintaining their distinction from physics or chemistry.
Key Results and Insights
Improved Model Fit
The CTM consistently outperformed LDA in terms of predictive power. When measured by log-likelihood on held-out data (a standard metric for how well a model explains unseen documents), the CTM provided a significantly better fit. By understanding which topics "belong" together, the model can more accurately predict the word distributions of new documents.
Qualitative Discovery and Topology
Beyond statistical metrics, the CTM provides a superior exploratory experience. While LDA produces a flat list of topics, the CTM reveals the "topology" of the data. In the Science corpus, the CTM successfully mapped "Genetics" as a central hub connected to "Evolution," "Medicine," and "Cell Biology." This relational structure allows researchers to see not just what the topics are, but how they interact to form the broader scientific landscape.
Conclusion and Future Directions
The Correlated Topic Model represents a significant evolution in the statistical analysis of discrete data. While it is more computationally demanding than LDA—due to the overhead of modeling a $K \times K$ covariance matrix—the gains in interpretability and predictive accuracy are substantial.
This work lays the groundwork for further extensions, such as Dynamic Topic Models (which track how these correlations evolve over time) or models that incorporate metadata like citations and authorship. By moving away from the assumption of independence, the CTM provides a more faithful reflection of the interconnected nature of human knowledge.
Summary updated