Visualizing Topics with Multi-Word Expressions

David M. Blei, John LaffertyarXiv 2009 · DOI · arXiv

Abstract

We describe a new method for visualizing topics, the distributions over terms that are automatically extracted from large text corpora using latent variable models. Our method finds significant $n$-grams related to a topic, which are then used to help understand and interpret the underlying distribution. Compared with the usual visualization, which simply lists the most probable topical terms, the multi-word expressions provide a better intuitive impression for what a topic is "about." Our approach is based on a language model of arbitrary length expressions, for which we develop a new methodology based on nested permutation tests to find significant phrases. We show that this method outperforms the more standard use of $χ^2$ and likelihood ratio tests. We illustrate the topic presentations on corpora of scientific abstracts and news articles.

Introduction

Topic modeling is a cornerstone of unsupervised machine learning, allowing researchers to summarize massive document collections into "topics"—mathematical distributions over words. However, most models rely on the "bag-of-words" assumption, treating words as exchangeable units and ignoring their sequential order. While computationally efficient, this simplification often results in cryptic word lists that are difficult to interpret. For instance, a topic about physics might list "phase" and "transition" separately, losing the specific meaning of the compound term. Previous attempts to incorporate phrases directly into generative models often suffered from high computational overhead or statistical instability.

Methodology: The Turbo Topics Approach

The authors introduce Turbo Topics, a post-processing strategy that enhances the output of standard unigram models like Latent Dirichlet Allocation (LDA). Instead of modifying the underlying generative process, Turbo Topics "recycles" the model's posterior distributions.

The process begins by annotating every word in the corpus with its most likely topic assignment. These sequences of topic-specific words are then treated as a separate "language." To identify significant multi-word expressions, the method employs a recursive permutation test. Unlike traditional $\chi^2$ or likelihood ratio tests, which rely on asymptotic assumptions that often fail with sparse text data, permutation tests are distribution-free. By shuffling the data to determine the probability of a phrase appearing by chance, the method ensures statistical conservativeness.

Comparison of standard LDA unigram lists versus Turbo Topic multi-word expressions, alongside the document annotation process.

This approach handles polysemy effectively. Because words are labeled based on their document context before phrase extraction, "fly" can be correctly identified as part of "fruit fly" in a biology context and "frequent flyer" in a travel context.

Evaluation and Results

The performance of Turbo Topics was evaluated using simulated data, news articles from the Huffington Post, and scientific abstracts from the Physics arXiv.

Quantitative Performance

On simulated data generated via a Chinese Restaurant Process (CRP), the nested permutation test significantly outperformed standard statistical tests. As shown in the F-measure comparisons, the permutation-based approach maintains high precision and recall even as data sparsity increases, whereas $\chi^2$ and likelihood ratio tests tend to over-identify phrases in smaller samples.

F-measure comparison across different statistical tests (Chi-squared, Likelihood ratio, and Permutation tests) for varying p-values and sample sizes.

Qualitative Clarity

In real-world datasets, the improvement in interpretability is stark. In the Physics arXiv corpus, where a standard model might list "black" and "holes" as separate high-probability terms, Turbo Topics successfully identifies "black hole mass" and "supermassive black holes." Similarly, for news data, the method moves beyond generic terms like "court" to identify specific entities such as "the california supreme court."

Conclusion

Turbo Topics demonstrates that significant gains in topic interpretability can be achieved without abandoning efficient bag-of-words models. By using latent variables as a filter for sophisticated linguistic analysis, the method provides a "turbo-charged" visualization that is both statistically robust and intuitively readable. Future work may explore more computationally efficient procedures for multiple hypothesis testing and the application of discovered phrases to information retrieval tasks.

Summary updated

Abstract

We describe a new method for visualizing topics, the distributions over terms that are automatically extracted from large text corpora using latent variable models. Our method finds significant $n$-grams related to a topic, which are then used to help understand and interpret the underlying distribution. Compared with the usual visualization, which simply lists the most probable topical terms, the multi-word expressions provide a better intuitive impression for what a topic is "about." Our approach is based on a language model of arbitrary length expressions, for which we develop a new methodology based on nested permutation tests to find significant phrases. We show that this method outperforms the more standard use of $χ^2$ and likelihood ratio tests. We illustrate the topic presentations on corpora of scientific abstracts and news articles.
BibTeX
@misc{blei2009visualizing,
  author = {David M. Blei and John Lafferty},
  title = {{Visualizing Topics with Multi-Word Expressions}},
  year = {2009},
  eprint = {0907.1013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0907.1013},
  howpublished = {arXiv},
}
Figure