TL;DR
Researchers have developed the IBP Compound Dirichlet Process (ICD), a new Bayesian nonparametric model that fixes a major flaw in standard topic modeling: the assumption that rare topics must also be "thin" topics. By combining the Indian Buffet Process (IBP) with Dirichlet distributions, their Focused Topic Model (FTM) allows rare, niche topics to dominate specific documents, leading to better data fit and more compact representations than the industry-standard Hierarchical Dirichlet Process (HDP).
What Problem Does This Paper Address?
In the world of automated text analysis, the Hierarchical Dirichlet Process (HDP) is a heavyweight. It allows a model to automatically discover an unknown number of topics within a collection of documents. However, the HDP harbors a "hidden assumption": a topic’s overall popularity across the entire library is positively correlated with how much space it takes up in any single document.
This is often untrue in the real world. Consider a niche topic like "baseball." It might be rare across a general news corpus (low global prevalence), but in the few articles where it does appear, it likely accounts for nearly 100% of the words (high local proportion). The HDP struggles with this; it forces rare topics to have small proportions and common topics to have large ones. The authors aim to "decouple" these two features so that a model can be truly "focused."
Methods
The authors introduce the IBP Compound Dirichlet Process (ICD). To understand how it works, it helps to look at its two parents:
- Indian Buffet Process (IBP): A prior used to select a subset of features (topics) for each data point from an infinite pool. It handles prevalence—which topics show up where.
- Dirichlet Process (DP): Traditionally used to handle proportions—how much of each topic is present.
In the ICD, the IBP acts as a "spike and slab" filter. For each document, the IBP selects a finite "focus" of topics. Then, independent Gamma-distributed variables determine the relative weights of these selected topics. This structure ensures that the decision of whether a topic appears is mathematically separated from how much of that topic appears.
The resulting application for text is called the Focused Topic Model (FTM). Because the math involves integrating over an infinite combinatorial space, the authors developed a specialized collapsed Gibbs sampler using a second-order Taylor expansion to make inference computationally feasible.
Datasets
The model was evaluated on three standard text corpora:
- PNAS: 1,766 abstracts from the Proceedings of the National Academy of Sciences (1991–2001).
- 20 Newsgroups: A collection of 1,000 randomly selected articles from the famous newsgroup dataset.
- Reuters-21578: 2,000 randomly selected documents from the Reuters financial news wire.
Key Insights and Ideas
- The "Focus" Effect: By using the IBP, the model forces each document to pick a specific subset of topics. This mirrors human writing better than the HDP, which tends to "smear" a little bit of every common topic into every document.
- Independence is Power: By making topic weights independent of their global frequency, the FTM can identify "heavy-hitting" niche topics that the HDP would either ignore or dilute.
- Compactness: The FTM actually prefers a more "compact" representation of the data. It tends to find fewer total topics across the corpus but assigns more topics per document compared to the HDP.
- Word-Topic Distinctness: In the FTM, individual words are associated with fewer topics on average. This suggests the topics discovered are more semantically distinct and less "muddy."
Key Results
The FTM demonstrated clear advantages over the HDP across multiple quantitative metrics:
- Better Generalization: The FTM consistently achieved lower perplexity (a measure of how well the model predicts unseen data) across all three datasets compared to the HDP.
- Successful Decoupling: Quantitative analysis confirmed that the correlation between topic frequency (prevalence) and topic proportion was significantly lower in the FTM than in the HDP, proving the model works as intended.
- Efficiency: Despite the more complex mathematical framework, the FTM's runtime was comparable to the HDP in practice because its more compact representation required fewer total topics to be tracked during computation.
Limitations and Future Work
The authors note that while the FTM removes the correlation between global and local probabilities, it doesn't yet account for correlations between topics (e.g., "baseball" and "stadium" appearing together). They suggest that integrating topic correlation structures (like those in Correlated Topic Models) into the FTM framework is a promising next step.
Additionally, while this paper focused on text, the ICD is a general-purpose prior. The authors suggest it could be highly effective in other areas where the HDP is currently used, such as Hidden Markov Models, where certain states might be rare globally but highly persistent once entered.
Summary updated