Topic Intrusion for Automatic Topic Model Evaluation

Shraey Bhatia, Jey Han Lau, Timothy BaldwinEMNLP 2018 · OpenReview

Introduction

Evaluating topic models like Latent Dirichlet Allocation (LDA) has traditionally relied on metrics such as perplexity or topic coherence. While topic coherence measures the internal semantic consistency of a topic (e.g., "space, orbit, NASA"), it often fails to assess document-level utility. A model can generate coherent topics that are entirely irrelevant to the specific document being analyzed.

The gold standard for measuring this utility is the topic intrusion task, where human annotators are asked to identify an "intruder" topic among three relevant ones. However, manual testing is prohibitively expensive and slow. This paper introduces a neural-network-based approach to automate the topic intrusion task, providing a scalable and nuanced evaluation metric that aligns closely with human judgment.

Methodology

The proposed system uses a deep learning architecture designed to "rank" topics based on their relevance to a specific document.

Neural network architecture for automated topic intrusion.

As illustrated in the architecture diagram, the model processes a Document Matrix and a Topic Matrix through a Convolutional Neural Network (CNN). Convolutional filters extract feature maps, which are then processed via max pooling to create fixed-length vector embeddings. To enhance performance, the authors incorporate External Features using Okapi BM25, which provides a corpus-wide relevance score for topic words against the document context. These representations are concatenated and passed through dense hidden layers to a sigmoid output.

A key contribution is the Normalized Sigmoid Score ($nss$). Unlike previous binary classification attempts, $nss$ provides a continuous value between 0 and 1. This allows the model to capture the "gray areas" of topic relevance, reflecting the inherent uncertainty often found in human judgments.

Experimental Setup

The model was evaluated on two diverse corpora: APNEWS (Associated Press articles) and BNC (British National Corpus). The researchers tested five distinct topic models, including adversarial models designed to exploit coherence metrics. Ground-truth data was established through thousands of human judgments collected via Amazon Mechanical Turk to ensure the automated metric could be validated against real-world performance.

Key Results

The automated $nss$ metric demonstrated a significant improvement over previous state-of-the-art methods across multiple dimensions.

Correlation of automated metrics with human gold standards on APNEWS and BNC datasets.

As shown in the performance plots, the $nss$ metric (yellow line) tracks the human gold standard ($mp_{GOLD}$, blue line) much more accurately than the baseline $mp_{ORIG}$. On the APNEWS dataset, the $nss$ score achieved a remarkably high correlation of $r = 0.92$ with human performance.

Furthermore, the model proved robust across domains. In cross-domain evaluations (e.g., training on BNC and testing on APNEWS), the Mean Absolute Error (MAE) remained consistently low, often reducing error by 50% or more compared to baselines. This suggests the network learns generalizable relationships between text and topics rather than just memorizing specific keywords from a single dataset.

Conclusion

By automating the topic intrusion task, this research provides a more reliable way to evaluate the descriptive power of topic models. The $nss$ metric bridges the gap between internal coherence and external utility, offering a scalable alternative to human evaluation. Future work will focus on simplifying the pipeline by reducing reliance on external IR systems and unifying coherence and intrusion into a single comprehensive metric.

Summary updated

BibTeX
@inproceedings{bhatia2018topic,
  author = {Shraey Bhatia and Jey Han Lau and Timothy Baldwin},
  title = {{Topic Intrusion for Automatic Topic Model Evaluation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=HyWSWffdZr},
}

Loading related content...

Figure