TL;DR
This paper introduces a novel, manually annotated dataset of 4,320 tweets to analyze COVID-19 vaccine sentiment within Nigerian cyberspace. By benchmarking classical machine learning models against fine-tuned large language models (LLMs), the authors demonstrate that transformer-based models like XLM-RoBERTa excel at capturing nuanced public sentiment, even when dealing with code-mixed regional languages.
What Problem Does This Paper Address?
Public sentiment is a critical driver of vaccine acceptance and the overall success of public health initiatives. While social media sentiment analysis is widely used to gauge public opinion, most existing datasets and models focus heavily on Western contexts and standard English.
There is a significant gap in high-quality, localized datasets capturing public discourse in African contexts. In Nigeria, online discussions frequently feature Nigerian English code-mixed with local languages and regional slang. Standard sentiment analysis tools fail to capture these nuances, making it difficult for public health officials to accurately monitor vaccine hesitancy or misinformation within the region.
Methods
The authors developed a localized pipeline consisting of data collection, manual annotation, and model benchmarking:
- Data Collection: Tweets were scraped using specific keywords (e.g., vaccin, pfizer, moderna, microchips, infertility*) and filtered strictly by Nigerian location IDs.
- Manual Annotation: A rigorous three-annotator pipeline classified tweets into Positive, Negative, Neutral, or Indeterminate. Disagreements were resolved using a strict majority-voting and adjudication protocol.
- Modeling: The authors benchmarked two distinct paradigms:
- Classical Machine Learning: Support Vector Machines (SVM), Logistic Regression, Naive Bayes, K-Nearest Neighbours (KNN), and Decision Trees.
- Fine-tuned LLMs: BERT-base, Afro-XLMR (a language model adapted for African languages), and XLM-RoBERTa-base.
Datasets
The primary contribution is a newly curated, novel dataset consisting of 4,320 preprocessed and manually annotated tweets from Nigerian cyberspace, spanning from November 1, 2020, to June 30, 2022. The dataset captures real-world public discourse surrounding major vaccine brands (AstraZeneca, Pfizer, Moderna, Johnson & Johnson) alongside prevalent conspiracy theories.
Key Insights and Ideas
- Dominant Neutrality: Over half of the analyzed tweets (54.06%) were neutral. This indicates that a large portion of the digital population used social media to share general updates or news rather than expressing strong polarization.
- Context is King: Fine-tuned LLMs consistently outperformed classical machine learning models. Because LLMs leverage transformer architectures trained to grasp contextual relationships, they are far better suited to handle the linguistic complexities of Nigerian code-mixed English.
- Cross-Lingual Transferability: Interestingly, the authors found that fine-tuning a massively multilingual model (like XLM-RoBERTa) yields highly competitive results on localized datasets, even if the model was not explicitly pre-trained on the specific regional code-mixed dialects.
- Linguistic Divergence: Word cloud analysis revealed distinct emotional markers. Positive tweets heavily relied on words like safe, free, protect, and help, whereas negative sentiment was dominated by risk-associated terms like kill, pain, conspiracy, and infected.
Key Results
-
Sentiment Distribution: The dataset breakdown revealed 54.06% Neutral, 27.20% Positive, and 18.74% Negative sentiments.
Sentiment distribution of tweets in the dataset. -
Classical ML Performance: Among the traditional models, SVM and Logistic Regression achieved the highest F1-score of 0.65, though SVM showed superior overall accuracy (0.66). Naive Bayes performed the poorest with an F1-score of 0.45.
Performance metrics of classical machine learning models. -
LLM Performance: Fine-tuned LLMs set the performance ceiling. XLM-RoBERTa base achieved the top performance with an F1-score of 0.71 and an accuracy of 0.71. BERT base followed closely with an F1-score of 0.70, demonstrating the robust capability of transformer architectures on small, specialized datasets.
Limitations and Future Work
The authors note two primary limitations:
- Dataset Scale: Comprising fewer than 4,400 samples, the dataset is relatively small, which can restrict the generalization capabilities of deeply parameterized deep learning models.
- Label Imbalance: The heavy skew toward neutral tweets introduces a class imbalance that can bias model predictions.
Future work will focus on expanding the dataset size, exploring more advanced text preprocessing techniques, and conducting deeper hyperparameter optimization to squeeze more performance out of the fine-tuned language models.
Summary updated