Deciphering Vaccine Sentiments: Transformer Models in Social Media Analysis

Shriram M. SFIRE (Working Notes) 2023 · OpenReview

Introduction

Vaccine hesitancy is a complex public health challenge, often amplified by the rapid spread of diverse concerns and misinformation on social media platforms. Unlike simple sentiment analysis, understanding vaccine-related discourse requires identifying specific, overlapping concerns. A single tweet might express anxiety regarding pharmaceutical profits, government mandates, and potential side effects simultaneously. This study addresses this as a multi-label classification problem, evaluating the efficacy of transformer-based models in categorizing nuanced anti-vaccine sentiments.

Methodology

The research team from Sri Sivasubramaniyanadar College of Engineering compared three prominent transformer architectures: BERT, RoBERTa, and XLNet.

Preprocessing

To handle the inherent noise of social media data, a rigorous preprocessing pipeline was implemented. This included the removal of user tags (@), HTML tags, brackets, and emojis. Text was further normalized through stop-word removal and lemmatization to reduce words to their linguistic base forms.

Model Architectures and Training

  1. BERT (Bidirectional Encoder Representations from Transformers): Served as the baseline, utilizing a Masked Language Model (MLM) to capture bidirectional context.
  2. RoBERTa (Robustly Optimized BERT approach): An optimized version of BERT that employs dynamic masking and removes the next-sentence prediction task for improved robustness.
  3. XLNet: An autoregressive model using a permutation-based approach to capture bidirectional context, designed to handle long-range dependencies more effectively.

For each model, a classification head with 12 output units and a sigmoid activation function was used to facilitate multi-label prediction. The specific training configurations used for these models are detailed below.

Table 1: Hyperparameter configurations and training accuracy for the transformer models.

Dataset

The study utilized the CAVES dataset from the AISoMe track at FIRE 2023. The training set consisted of 9,921 anti-vaccine tweets from 2020–2021, while 486 records were reserved for testing. The models were tasked with identifying 12 unique categories of concern, including Unnecessary, Mandatory, Pharma, Conspiracy, Political, Country, Rushed, Ingredients, Side-effect, Ineffective, Religious, and None.

Results

Performance was evaluated using Macro F1-Score and Jaccard Score to account for class imbalance and label overlap in the multi-label task.

Table 2: Performance comparison of BERT, RoBERTa, and XLNet on the test set.

While BERT achieved the highest training accuracy (99.2%), RoBERTa demonstrated superior generalization on the test set, achieving the highest Jaccard Score (0.57) and matching BERT's Macro F1-Score (0.57). XLNet trailed both encoder-based models significantly in this specific task, likely due to the nature of short-form social media text.

Discussion and Conclusion

The results underscore the importance of the self-attention mechanism in parsing multi-faceted social media posts. RoBERTa’s slight edge over BERT suggests that dynamic masking and more robust training recipes are better suited for the "messy" linguistic variations of Twitter data.

However, the gap between high training accuracy and modest test scores (Macro-F1 of 0.57) highlights the difficulty of generalizing across the evolving landscape of social media discourse. Future work will focus on adversarial training to improve model resilience against sarcasm and misspellings, as well as data augmentation techniques to overcome the limitations of the current dataset size.

Summary updated

BibTeX
@inproceedings{s2023deciphering,
  author = {Shriram M. S},
  title = {{Deciphering Vaccine Sentiments: Transformer Models in Social Media Analysis}},
  booktitle = {FIRE (Working Notes)},
  year = {2023},
  url = {https://openreview.net/forum?id=LAY34Xm3X9},
}
Figure