Introduction
Traditional topic models like Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Processes (HDP) have long been the workhorses of computational social science. However, these models often struggle with the inherent noise, brevity, and rapid evolution of social media data. While Neural Topic Models (NTMs) based on Variational Autoencoders (VAEs) offer a promising alternative, they frequently suffer from "mode collapse"—a state where the model learns only a few redundant topics. This paper presents a comparative analysis, demonstrating how a diversity-regularized NTM can significantly outperform classical methods in analyzing large-scale COVID-19 Twitter discourse.
Addressing the "Mode Collapse" in Neural Topic Models
The researchers focused on the Embedded Topic Model (ETM), which maps words and topics into a shared embedding space. To overcome the limitations of standard NTMs, they introduced three critical enhancements:
- Domain-Specific Pretraining: Initializing the model with Word2Vec embeddings trained directly on the COVID-19 corpus.
- Topic Diversity Regularization (TD): A novel penalty term based on the Total Variation (TV) distance between topic distributions. This forces the model to learn distinct semantic clusters rather than converging on a few "safe" global keywords.
- Comparative Benchmarking: Evaluating the enhanced ETM against HDP, a non-neural baseline that is favored for its ability to automatically determine the number of topics.
Dataset and Preprocessing
The study utilized a massive dataset of over 15 million tweets related to the COVID-19 pandemic, spanning January to April 2020. For the analysis, the authors sampled 10,000 tweets per day, resulting in a training set of approximately 875,000 tweets and a test set of 97,000. Standard preprocessing steps—including tokenization, lemmatization, and stop-word removal—were applied to ensure data quality.
Quantitative Performance
The results clearly indicate that the proposed ETM + W2V + TD model provides a superior fit for social media data. As shown in the performance metrics, the neural approach achieves significantly lower perplexity and higher topic diversity compared to the HDP baseline.
Specifically, the addition of the TD regularization term boosted the Topic Gap (a measure of diversity) from 0.12 to 0.55, effectively solving the mode collapse problem that plagues standard ETM implementations.
Qualitative Insights: Cleaner and More Diverse Topics
Beyond automated metrics, the qualitative difference between neural and non-neural models is striking. Traditional models like HDP often produce "noisy" topics dominated by ubiquitous terms like "coronavirus" or "covid," which appear across almost every discovered theme. In contrast, the diversity-regularized ETM filters out this background noise to reveal specific, actionable sub-themes.
As illustrated in the keyword comparison, the ETM + W2V + TD model identifies distinct political sub-themes (e.g., state-level governance with "governor, california, florida") and social impacts (e.g., community response and emotional support) that are obscured in the HDP output. This level of granularity allows social scientists to track shifts in public discourse—such as the transition from global news updates to local containment measures—with much higher precision.
Conclusion and Future Directions
This research demonstrates that with proper regularization and pretraining, Neural Topic Models are not just viable but superior tools for large-scale social media analysis. By addressing mode collapse, the authors have bridged the gap between advanced machine learning and practical social science applications. Future work will focus on automating hyperparameter selection for NTMs and testing these models on less dynamic datasets to further validate their generalizability.
Summary updated