The Sensitivity of Topic Coherence Evaluation to Topic Cardinality

Jey Han Lau, Timothy Baldwin

Introduction

In the evaluation of topic models, researchers typically examine the top $N$ words of a generated topic to assess its "coherence." This parameter $N$, known as cardinality, is often chosen arbitrarily (commonly 5 or 10). However, this study reveals that the choice of $N$ significantly impacts both human perception and the performance of automated evaluation metrics. As $N$ increases, automated metrics tend to diverge from human judgment. To address this, the authors propose an aggregation strategy that combines scores across multiple cardinalities for a more robust evaluation.

Methodology

The study investigates the sensitivity of two primary evaluation frameworks to changes in cardinality ($N \in \{5, 10, 15, 20\}$):

  1. Automated Word Intrusion: A supervised method using word association features (e.g., NPMI) to identify a manually injected "intruder" word. High detection accuracy indicates a coherent topic.
  2. NPMI (Normalized Pointwise Mutual Information): An unsupervised metric calculating the average pairwise association among the top-$N$ words.

The researchers introduced an aggregation strategy, averaging system scores and human ratings across all tested cardinalities to stabilize the final correlation results.

Datasets and Human Benchmarking

The experiments utilized two large-scale English corpora: WIKI (3.3M Wikipedia articles) and NEWS (1.2M New York Times articles).

To establish a ground truth, the authors collected human coherence judgments via Amazon Mechanical Turk. Participants rated 600 LDA-generated topics on a 3-point scale across the four cardinality settings. This dataset reveals how human perception of a topic's "essence" shifts as more words are presented.

Key Insights: The Dilution Effect and Correlation Decay

A primary finding is the "Dilution Effect": human ratings of topic coherence systematically decrease as $N$ increases. A topic that appears highly focused with 5 words often seems cluttered or less coherent when 20 words are shown.

Mean human coherence ratings across different cardinalities (N) for WIKI and NEWS domains.

Furthermore, the correlation between automated metrics and human ratings degrades as cardinality increases. Current metrics are more effective at mimicking human intuition for shorter word lists than for longer ones.

The Case for Aggregation

The study's most significant contribution is the effectiveness of aggregation. By averaging scores across multiple values of $N$, the researchers achieved correlations that were higher and more stable than those of any single cardinality. This approach effectively mitigates the noise associated with selecting an arbitrary cutoff.

Pairwise correlations of human coherence ratings across different cardinalities (5, 10, 15, 20).

The scatter plots above illustrate the relationship between human ratings at different cardinalities. While there is a clear positive correlation, the variance suggests that a single $N$ may not capture the full picture of topic quality.

Key Results

  • Human Perception: Mean coherence ratings dropped from approximately 2.45 at $N=5$ to 2.34 at $N=20$ across both domains.
  • Metric Performance: In the NEWS domain, the correlation for the Word Intrusion task fell from 0.65 ($N=5$) to 0.47 ($N=20$).
  • Stability: Aggregating across all $N$ values yielded a correlation of 0.65 for Word Intrusion and 0.71 for NPMI on the NEWS dataset, matching or exceeding the best individual settings.

Conclusion and Future Directions

The study highlights a practical limit to human evaluation; as $N$ exceeds 20, the task becomes increasingly difficult for human raters. While the research focused on LDA, the findings suggest that future evaluations of any topic modeling approach should consider multiple cardinalities. The proposed aggregation method offers a simple, unsupervised way to improve model selection and evaluation reliability in real-world NLP pipelines.

Summary updated

Abstract

When evaluating the quality of topics generated by a topic model, the convention is to score topic coherence — either manually or automatically — using the top-N topic words. This hyper-parameter N , or the cardinality of the topic, is often overlooked and selected arbitrarily. In this paper, we investigate the impact of this cardinality hyper-parameter on topic coherence evaluation. For two automatic topic coherence methodologies, we observe that the correlation with human ratings decreases systematically as the cardinality increases. More interestingly, we find that performance can be improved if the system scores and human ratings are aggregated over several topic cardinalities before computing the correlation. In contrast to the standard practice of using a fixed value of N (e.g. N = 5 or N = 10), our results suggest that calculating topic coherence over several different cardinalities and averaging results in a substantially more stable and robust evaluation. We release the code and the datasets used in this research, for reproducibility. 1
BibTeX
@inproceedings{lau2016sensitivity,
  author = {Jey Han Lau and Timothy Baldwin},
  title = {{The Sensitivity of Topic Coherence Evaluation to Topic Cardinality}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/n16-1057},
}
Figure