Introduction
In the evaluation of topic models, researchers typically examine the top $N$ words of a generated topic to assess its "coherence." This parameter $N$, known as cardinality, is often chosen arbitrarily (commonly 5 or 10). However, this study reveals that the choice of $N$ significantly impacts both human perception and the performance of automated evaluation metrics. As $N$ increases, automated metrics tend to diverge from human judgment. To address this, the authors propose an aggregation strategy that combines scores across multiple cardinalities for a more robust evaluation.
Methodology
The study investigates the sensitivity of two primary evaluation frameworks to changes in cardinality ($N \in \{5, 10, 15, 20\}$):
- Automated Word Intrusion: A supervised method using word association features (e.g., NPMI) to identify a manually injected "intruder" word. High detection accuracy indicates a coherent topic.
- NPMI (Normalized Pointwise Mutual Information): An unsupervised metric calculating the average pairwise association among the top-$N$ words.
The researchers introduced an aggregation strategy, averaging system scores and human ratings across all tested cardinalities to stabilize the final correlation results.
Datasets and Human Benchmarking
The experiments utilized two large-scale English corpora: WIKI (3.3M Wikipedia articles) and NEWS (1.2M New York Times articles).
To establish a ground truth, the authors collected human coherence judgments via Amazon Mechanical Turk. Participants rated 600 LDA-generated topics on a 3-point scale across the four cardinality settings. This dataset reveals how human perception of a topic's "essence" shifts as more words are presented.
Key Insights: The Dilution Effect and Correlation Decay
A primary finding is the "Dilution Effect": human ratings of topic coherence systematically decrease as $N$ increases. A topic that appears highly focused with 5 words often seems cluttered or less coherent when 20 words are shown.
Furthermore, the correlation between automated metrics and human ratings degrades as cardinality increases. Current metrics are more effective at mimicking human intuition for shorter word lists than for longer ones.
The Case for Aggregation
The study's most significant contribution is the effectiveness of aggregation. By averaging scores across multiple values of $N$, the researchers achieved correlations that were higher and more stable than those of any single cardinality. This approach effectively mitigates the noise associated with selecting an arbitrary cutoff.
The scatter plots above illustrate the relationship between human ratings at different cardinalities. While there is a clear positive correlation, the variance suggests that a single $N$ may not capture the full picture of topic quality.
Key Results
- Human Perception: Mean coherence ratings dropped from approximately 2.45 at $N=5$ to 2.34 at $N=20$ across both domains.
- Metric Performance: In the NEWS domain, the correlation for the Word Intrusion task fell from 0.65 ($N=5$) to 0.47 ($N=20$).
- Stability: Aggregating across all $N$ values yielded a correlation of 0.65 for Word Intrusion and 0.71 for NPMI on the NEWS dataset, matching or exceeding the best individual settings.
Conclusion and Future Directions
The study highlights a practical limit to human evaluation; as $N$ exceeds 20, the task becomes increasingly difficult for human raters. While the research focused on LDA, the findings suggest that future evaluations of any topic modeling approach should consider multiple cardinalities. The proposed aggregation method offers a simple, unsupervised way to improve model selection and evaluation reliability in real-world NLP pipelines.
Summary updated