TL;DR
This paper introduces and applies two novel automated metrics to evaluate the quality of topics generated by three popular topic modeling techniques: LDA, LSA (SVD), and NMF. They find that while LDA and NMF often produce more coherent individual topics, LSA (SVD) excels at creating a compact representation that is more useful for downstream tasks like document classification and better aligns with human judgments of word similarity.
What Problem Does This Paper Address?
Topic modeling techniques like Latent Dirichlet Allocation (LDA), Latent Semantic Analysis (LSA), and Non-negative Matrix Factorization (NMF) are powerful tools for uncovering hidden thematic structures in large text corpora. However, evaluating the "quality" of the topics they produce has been a persistent challenge. Traditional methods often rely on human judgment, which is time-consuming and subjective, or intrinsic measures like perplexity, which don't always correlate well with human interpretations of topic interpretability. This paper aims to bridge this gap by providing automated, reliable metrics that can assess topic coherence and usefulness across different topic modeling approaches, helping researchers choose the best model for their specific application.
Methods
The authors employ two automated coherence measures: the UCI measure and the UMass measure. Both metrics aim to quantify how semantically related the top words within a topic are.
- UCI measure: This metric uses pointwise mutual information (PMI) between word pairs, calculated from co-occurrence frequencies in an external corpus (Wikipedia). It's considered an "external" comparison.
- UMass measure: This metric also uses PMI but calculates word co-occurrences directly from the training corpus itself. This makes it a more "intrinsic" measure, assessing how well the model has captured the corpus's own semantic relationships.
They also introduce aggregate measures to evaluate an entire topic model, rather than just individual topics. These include the average coherence of all topics and the entropy of topic coherence, which helps differentiate between models that produce uniformly good topics versus those with a wide range of quality.
For evaluation, they compare LDA, LSA (specifically using Singular Value Decomposition - SVD), and NMF. They also conduct experiments on word similarity tasks and document classification to assess the practical utility of the learned topics.
Datasets
- Training and Evaluation: The primary dataset used is the New York Times Annotated Corpus from 2003, comprising 92,600 articles. After preprocessing (stop word removal, minimum word frequency filtering), 35,836 unique tokens remained.
- External Corpus for UCI Measure: WaCkypedia corpus was used for calculating word co-occurrence probabilities for the UCI coherence measure.
- Word Similarity Tasks:
- Rubenstein and Goodenough (1965) dataset (65 word pairs).
- Finkelstein et al. (2002) dataset (353 word pairs).
Key Insights and Ideas
- Smoothing Factor Matters: The choice of the smoothing factor ($\epsilon$) in the coherence measures significantly impacts results, particularly for NMF and SVD. A higher $\epsilon$ can artificially inflate scores for incoherent topics, suggesting that smaller values are generally preferable.
- Coherence vs. Usefulness: High topic coherence doesn't always translate to better performance in downstream tasks. LSA (SVD), despite often having lower coherence scores, proved superior in word similarity judgments and document classification.
- LDA and NMF Strengths: LDA and NMF tend to learn more descriptive and coherent individual topics, making them potentially better for applications where human interaction with specific topics is key.
- LSA (SVD) as a Compact Representation: LSA (SVD) excels at creating a compact, generalized semantic representation of documents and words, which is highly effective for tasks requiring a broad understanding of relationships across the corpus.
- Topic Importance in Classification: Not all topics are equally informative for classification. The most coherent topics are not necessarily the most useful; rather, topics with moderate coherence and higher feature importance in classification models drive decision-making.
Key Results
- Average Coherence: LDA and NMF generally performed similarly and better than SVD in terms of average topic coherence, especially with the UMass measure. However, the UCI measure revealed more separation, highlighting NMF's potential for learning very low-quality topics.
- Word Similarity: SVD significantly outperformed LDA and NMF on the Rubenstein & Goodenough test and was the best model on the Finkelstein et al. test, indicating its topics provide a more accurate approximation of human similarity judgments.
- Document Classification: SVD achieved slightly, but statistically significantly, higher classification accuracy than NMF and LDA. Performance increased rapidly with the number of topics and plateaued under 50 topics.
- Topic Rank vs. Coherence: For classification, topics with moderate coherence and higher feature importance were more influential. The Spearman rank correlation between topic coherence and classification feature rank was highest for NMF but generally low for all models beyond 100 topics.
Limitations and Future Work
- Smoothing Factor Sensitivity: The authors highlight the sensitivity of coherence measures to the smoothing factor ($\epsilon$), suggesting that future work should focus on robust methods for selecting this parameter.
- Human Judgment Alignment: While the automated metrics show promise, the paper acknowledges that coherence scores don't always indicate better representation of distributional information, as seen with SVD's performance in word similarity tasks.
- Topic Interpretability: The paper focuses on coherence and classification utility. Future work could explore more direct measures of human interpretability.
- Model Flexibility: The authors suggest that for applications requiring human interaction with learned topics, the flexibility of LDA and its coherence advantages warrant strong consideration.
- Code Availability: The authors mention that all code for this work will be made available through an open-source project.
Summary updated