Detecting COVID-19 Conspiracy Theories with Transformers and TF-IDF

Haoming Guo, Tianyi Huang, Huixuan Huang, Mingyue Fan, Gerald Friedland

Abstract

This study investigates the detection of COVID-19 conspiracy theories on Twitter by comparing traditional machine learning baselines with modern Transformer-based architectures. Using the MediaEval 2021 benchmark, researchers evaluated TF-IDF-based SVMs and Random Forests against BERT and RoBERTa models. The findings demonstrate that while pre-trained Transformers generally provide superior performance, specific architectural tuning and ensemble techniques can enable "from-scratch" Transformers to achieve high accuracy on niche, imbalanced datasets.

Introduction

The proliferation of misinformation and conspiracy theories on social media platforms presents significant real-world risks, particularly during global health crises. Unlike general fake news, conspiracy theories often rely on domain-specific, rapidly evolving vocabularies. This study addresses the challenge of classifying tweets into specific conspiracy categories using the MediaEval 2021 "FakeNews: Corona Virus and Conspiracies" benchmark.

Methodology

The researchers evaluated three primary approaches:

  1. TF-IDF Baselines: Support Vector Machines (SVM) and Random Forest (RF) classifiers were trained on TF-IDF vectorized text. Class weights were applied to mitigate the inherent class imbalance in the Twitter dataset.
  2. Transformer Architectures: BERT models were tested in both pre-trained and randomly initialized (from-scratch) states. The architecture utilized two fully connected layers on top of the hidden outputs for classification.
  3. Ensemble Strategies:
  4. Multi-layer Ensemble: Concatenating outputs from RoBERTa’s intermediate layers (8, 10, and 11) to capture diverse semantic signals.
  5. Multi-model Ensemble: Combining BERT and RoBERTa embeddings into a single joint classifier to create a more robust feature matrix.

Results and Discussion

The performance of the models was evaluated using the Matthews Correlation Coefficient (MCC), a metric robust to class imbalance.

Table 1: Matthews Correlation Coefficient (MCC) scores for various models across three conspiracy detection tasks.

As shown in Table 1, the results highlight several key insights:

  • Ensemble Robustness: The Multi-model Ensemble (BERT + RoBERTa) achieved the highest MCC for Task 1 (0.529) and Task 2 (0.567), demonstrating improved stability against individual model errors.
  • Pre-trained Dominance in Task 3: For Task 3, a standard pre-trained BERT model outperformed other configurations with an MCC of 0.525.
  • The "From-Scratch" Potential: Interestingly, the study found that by reducing hidden layer sizes and increasing learning rates, Transformers trained from scratch could achieve high accuracy (~90%) on specific tasks. This suggests that smaller, specialized models can be effective for niche domains without requiring massive pre-training.
  • Intermediate Layer Utility: Experiments with RoBERTa indicated that intermediate layers (8, 10, and 11) often provided stronger classification signals than the final output layer, particularly for nuanced tasks like stance detection.

Conclusion

While traditional TF-IDF models like SVM and Random Forest can achieve high raw accuracy, they struggle with minority classes in imbalanced datasets. Transformer-based models, especially when ensembled or specifically tuned for the domain, offer more robust detection of COVID-19 conspiracy theories. Future work will focus on better initialization methods for small-data Transformers and developing models capable of adapting to "trending vocabularies" in real-time.

Summary updated

BibTeX
@inproceedings{guo2021detecting,
  author = {Haoming Guo and Tianyi Huang and Huixuan Huang and Mingyue Fan and Gerald Friedland},
  title = {{Detecting COVID-19 Conspiracy Theories with Transformers and TF-IDF}},
  booktitle = {Mediaeval},
  year = {2021},
}
Figure