Benchmarking Neural Topic Models: An Empirical Study

Thanh-Nam Doan, Tuan-Anh HoangACL 2021

TL;DR

This paper provides a comprehensive empirical comparison of various neural topic modeling techniques against traditional methods like LDA. The authors find that while neural models excel at document representation and topic discovery, classical models often remain competitive or superior in modeling the corpus itself, highlighting the need for nuanced evaluation and selection of topic modeling tools.

What Problem Does This Paper Address?

Topic modeling aims to uncover latent thematic structures within large collections of text documents. Traditional methods, like Latent Dirichlet Allocation (LDA), have been the workhorses in this field for decades. More recently, neural topic models have emerged, promising to combine the strengths of deep learning (e.g., efficient inference, integration with pre-trained embeddings) with the probabilistic foundations of traditional models. However, the reported experimental results for these neural models are often inconsistent, making it difficult to assess their true advantages and limitations. This paper addresses this by conducting a systematic and extensive empirical study to benchmark popular neural topic models against each other and against classical approaches across various tasks and datasets.

Methods

The authors evaluate several neural topic models, primarily those based on encoder-decoder architectures, often leveraging variational autoencoders (VAEs). These models typically use an encoder to map documents to a latent topic representation and a decoder to reconstruct the document from this representation. The key variations among the neural models lie in their encoder architectures, the choice of variational distribution (e.g., Gaussian, Dirichlet, sparsemax), and the decoder design.

For comparison, they include two classical topic models: Non-negative Matrix Factorization (NMF) and Latent Dirichlet Allocation (LDA), with two variants of LDA (online variational inference and Gibbs sampling).

The evaluation is performed across three main tasks:

  1. Document Modeling: Measured by perplexity (both unseen documents and held-out words). Lower perplexity indicates a better fit to the data.
  2. Topic Discovery: Evaluated using topic coherence, specifically normalized point-wise mutual information (NPMI). Higher coherence suggests more interpretable topics.
  3. Document Representation: Assessed by the performance of the learned document topic vectors in a downstream classification task (using logistic regression). Higher F1 scores indicate better representation.

To ensure robust findings, experiments are repeated multiple times, and results are reported with mean and variance.

Datasets

The study utilizes four diverse datasets:

  • 20News: A collection of newsgroup documents, labeled by category.
  • W2E-content: Derived from news articles in the W2E dataset, labeled by topic.
  • W2E-title: Similar to W2E-content but uses article titles.
  • Web Snippets: Short text snippets collected from webpages, labeled by webpage category.

These datasets vary in document length and the number of labels, providing a good testbed for evaluating models across different scenarios.

Key Insights and Ideas

  • No Single Winner: None of the neural topic models consistently outperform the classical LDA across all datasets and metrics. This suggests that LDA remains a strong and competitive baseline.
  • Task-Specific Strengths: Neural models, particularly NVDM and NVCTM, show advantages in modeling documents (lower perplexity) and in document representation for downstream tasks (higher F1 scores). This is attributed to their ability to learn unnormalized topic vectors.
  • Topic Cohesion Trade-off: While NVDM and NVCTM are good at document modeling, they tend to underperform in topic discovery, yielding less cohesive topics compared to other models.
  • Classical Stability: Classical models (LDA, NMF) generally exhibit more stable performance, with smaller variances across experimental runs compared to neural models.
  • Scholar's Niche: The Scholar model is highlighted as being more suitable for uncovering cohesive, meaningful topics, likely due to its ability to incorporate metadata.

Key Results

  • Perplexity: NVDM and NVCTM generally achieve lower perplexity scores, indicating better document modeling.
  • Classification Performance (F1 Scores): NVDM and NVCTM also achieve significantly higher average F1 scores in downstream classification tasks, demonstrating superior document representation.
  • Topic Coherence: Classical models and some neural models like Scholar tend to achieve higher topic coherence scores, suggesting more interpretable topics.
  • Variability: Neural models, in general, show higher variance in their performance across runs compared to classical models.

Limitations and Future Work

The authors acknowledge that their study is not exhaustive and that other neural topic models exist. They plan to extend their work by:

  • Designing a fairer comparison framework for neural methods that might use different document representations (e.g., sequences of words).
  • Investigating other challenges in neural topic models, such as the posterior collapsing problem.

They also emphasize the importance of performing multiple runs on the same dataset for accurate evaluation and suggest that the choice of model should depend on the specific task requirements (e.g., document representation vs. topic interpretability).

Summary updated

BibTeX
@inproceedings{doan2021benchmarking,
  author = {Thanh-Nam Doan and Tuan-Anh Hoang},
  title = {{Benchmarking Neural Topic Models: An Empirical Study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure