Topic Modeling by Clustering Language Model Embeddings: Human Validation on an Industry Dataset

Anton Eklund, Mona ForsmanEMNLP 2022

TL;DR

This paper introduces STELLAR, a novel tool for human evaluation of topic models generated by clustering language model embeddings (NTM-CE). They demonstrate that NTM-CE can produce coherent topics on a real-world industry dataset, but highlight the critical need for human validation beyond automatic metrics, especially for production-ready applications.

What Problem Does This Paper Address?

In the realm of natural language processing, topic modeling is crucial for understanding large collections of text. A recent trend involves using pre-trained language models (PLMs) to generate document embeddings, which are then clustered to form topics. While this approach, termed Neural Topic Modeling by Clustering Embeddings (NTM-CE), shows promise for creating more coherent topics than traditional methods, evaluating its effectiveness is challenging. Automatic metrics like NPMI, often used to assess topic coherence, have been shown to not reliably emulate human judgment. Furthermore, there's a lack of human validation studies for NTM-CE, especially on industry-relevant datasets. This paper aims to bridge this gap by performing a human expert evaluation of an NTM-CE model on an industry dataset and introducing a tool to facilitate this process.

Methods

The authors employ a topic modeling pipeline that aligns with recent NTM-CE approaches, specifically building upon the BERTopic framework. The process involves:

  • Vectorization: Using a fine-tuned BERT model to generate document embeddings.
  • Dimension Reduction: Employing UMAP to reduce the dimensionality of these embeddings, aiming to reveal underlying patterns and reduce noise.
  • Clustering: Utilizing HDBSCAN for soft clustering of the reduced-dimension embeddings, allowing documents to belong to multiple clusters.

A key contribution is the development of STELLAR (Systematic Topic Evaluation Leveraging Lists of Articles). This interactive web-based tool allows users to:

  1. Get a visual overview of topics and their relationships.
  2. Quickly identify articles that don't fit a topic.
  3. Go beyond simple keywords to validate topics by reading article titles, keywords, and full text.

The human evaluation involved three experts who analyzed topics, identified "focus topics," and assessed the coherence of articles within those topics. They also rated how well the generated keywords represented the focus topic.

Datasets

The study uses a newly collected industry dataset consisting of 10,000 English online news articles published between May 29 and June 22, 2022. This dataset is characterized by its dynamic nature, featuring topics that are relevant for a short period, such as the Queen's Jubilee, cancelled flights, and Formula 1 racing. This dynamism makes it a good testbed for unsupervised topic models, as static classifiers would struggle to keep up.

Key Insights and Ideas

  • Human validation is paramount: Automatic metrics for topic coherence are insufficient. The study strongly advocates for human evaluation, especially for industry applications where trust and interpretability are key.
  • NTM-CE shows promise but isn't perfect: The NTM-CE approach, using BERT embeddings and HDBSCAN clustering, successfully generated coherent topics for a significant portion of the dataset.
  • "Incoherent" topics offer insights: The largest "incoherent" cluster, containing short articles, suggests potential issues with how short texts are handled or how padding artifacts might influence clustering. This warrants further investigation.
  • Topic granularity matters: The study highlights that the ideal topic size and balance might not be as strict as previously emphasized. The dynamic nature of clustering algorithms like HDBSCAN could be leveraged more effectively.
  • Keywords are helpful but not definitive: While keywords generated by the model received a "good" (2.8/4) rating, they were not considered "very good." This suggests room for improvement in keyword extraction or topic description methods.
  • STELLAR is a valuable evaluation tool: The developed tool effectively facilitates in-depth topic exploration and human evaluation, making the process more efficient and insightful.

Key Results

  • The topic modeling pipeline resulted in 63 clusters.
  • 52 out of 63 topics were deemed coherent by human evaluators (defined as Coh ≥ 0.8).
  • 57% of the articles in the dataset ended up in coherent topics.
  • Coherent topics had an average coherence score of 96%.
  • Evaluators agreed on the focus topic for 87% of topics, and for coherent topics, this agreement rose to 98%.
  • Evaluators agreed on overlapping articles for 95% of topics.
  • Keywords describing topics received an average rating of 2.8 out of 4 for representing the focus topic.

Limitations and Future Work

  • Small evaluator group: The study acknowledges that using only three evaluators might limit the strength of generalizable claims about NTM-CE. A more diverse group of evaluators is suggested to mitigate personal biases.
  • "Incoherent" topics: A significant portion of articles (43%) fell into incoherent topics. Further research is needed to understand and unravel the patterns within these topics, potentially by exploring stylistic or emotional characteristics.
  • Keyword improvement: While keywords were rated positively, there's room for improvement. Future work could focus on developing better keyword extraction methods or alternative ways to describe topics.
  • Dynamic topic evolution: The authors suggest exploring methods to detect emerging and disappearing topics over time, similar to work by Blei and Lafferty or Wang and McCallum.
  • STELLAR enhancements: Incorporating ideas from intrusion tasks could further improve the STELLAR tool for evaluating topic coherence.
  • Industry readiness: For NTM-CE to be fully production-ready for applications like contextual advertising, systems need to reliably identify narrow or trendy focus topics, which may still require human oversight.

Summary updated

BibTeX
@inproceedings{eklund2022topic,
  author = {Anton Eklund and Mona Forsman},
  title = {{Topic Modeling by Clustering Language Model Embeddings: Human Validation on an Industry Dataset}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure