Have you tried Neural Topic Models? Comparative Analysis of Neural and Non-Neural Topic Models with Application to COVID-19 Twitter Data

Andrew Bennett, Dipendra Misra, Nga ThanarXiv 2021 · arXiv

Introduction

Traditional topic models like Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Processes (HDP) have long been the workhorses of computational social science. However, these models often struggle with the inherent noise, brevity, and rapid evolution of social media data. While Neural Topic Models (NTMs) based on Variational Autoencoders (VAEs) offer a promising alternative, they frequently suffer from "mode collapse"—a state where the model learns only a few redundant topics. This paper presents a comparative analysis, demonstrating how a diversity-regularized NTM can significantly outperform classical methods in analyzing large-scale COVID-19 Twitter discourse.

Addressing the "Mode Collapse" in Neural Topic Models

The researchers focused on the Embedded Topic Model (ETM), which maps words and topics into a shared embedding space. To overcome the limitations of standard NTMs, they introduced three critical enhancements:

  1. Domain-Specific Pretraining: Initializing the model with Word2Vec embeddings trained directly on the COVID-19 corpus.
  2. Topic Diversity Regularization (TD): A novel penalty term based on the Total Variation (TV) distance between topic distributions. This forces the model to learn distinct semantic clusters rather than converging on a few "safe" global keywords.
  3. Comparative Benchmarking: Evaluating the enhanced ETM against HDP, a non-neural baseline that is favored for its ability to automatically determine the number of topics.

Dataset and Preprocessing

The study utilized a massive dataset of over 15 million tweets related to the COVID-19 pandemic, spanning January to April 2020. For the analysis, the authors sampled 10,000 tweets per day, resulting in a training set of approximately 875,000 tweets and a test set of 97,000. Standard preprocessing steps—including tokenization, lemmatization, and stop-word removal—were applied to ensure data quality.

Quantitative Performance

The results clearly indicate that the proposed ETM + W2V + TD model provides a superior fit for social media data. As shown in the performance metrics, the neural approach achieves significantly lower perplexity and higher topic diversity compared to the HDP baseline.

Table 1: Quantitative comparison of topic models across Perplexity (Perp), Coherence (Coh), Topic Gap (TG), and Contrast Score (CS).

Specifically, the addition of the TD regularization term boosted the Topic Gap (a measure of diversity) from 0.12 to 0.55, effectively solving the mode collapse problem that plagues standard ETM implementations.

Qualitative Insights: Cleaner and More Diverse Topics

Beyond automated metrics, the qualitative difference between neural and non-neural models is striking. Traditional models like HDP often produce "noisy" topics dominated by ubiquitous terms like "coronavirus" or "covid," which appear across almost every discovered theme. In contrast, the diversity-regularized ETM filters out this background noise to reveal specific, actionable sub-themes.

Table 2: Qualitative comparison of top words for "Politics" and "Social Impact" topics across different models.

As illustrated in the keyword comparison, the ETM + W2V + TD model identifies distinct political sub-themes (e.g., state-level governance with "governor, california, florida") and social impacts (e.g., community response and emotional support) that are obscured in the HDP output. This level of granularity allows social scientists to track shifts in public discourse—such as the transition from global news updates to local containment measures—with much higher precision.

Conclusion and Future Directions

This research demonstrates that with proper regularization and pretraining, Neural Topic Models are not just viable but superior tools for large-scale social media analysis. By addressing mode collapse, the authors have bridged the gap between advanced machine learning and practical social science applications. Future work will focus on automating hyperparameter selection for NTMs and testing these models on less dynamic datasets to further validate their generalizability.

Summary updated

Abstract

Topic models are widely used in studying social phenomena. We conduct a comparative study examining state-of-the-art neural versus non-neural topic models, performing a rigorous quantitative and qualitative assessment on a dataset of tweets about the COVID-19 pandemic. Our results show that not only do neural topic models outperform their classical counterparts on standard evaluation metrics, but they also produce more coherent topics, which are of great benefit when studying complex social problems. We also propose a novel regularization term for neural topic models, which is designed to address the well-documented problem of mode collapse, and demonstrate its effectiveness.
BibTeX
@misc{bennett2021have,
  author = {Andrew Bennett and Dipendra Misra and Nga Than},
  title = {{Have you tried Neural Topic Models? Comparative Analysis of Neural and Non-Neural Topic Models with Application to COVID-19 Twitter Data}},
  year = {2021},
  eprint = {2105.10165},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.10165},
}
Figure