Analyzing COVID-19 Vaccination Sentiments in Nigerian Cyberspace: Insights from a Manually Annotated Twitter Dataset

Ibrahim Said Ahmad, Lukman Jibril Aliyu, Abubakar Auwal Khalid, Saminu Muhammad Aliyu, Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Bala Mairiga Abduljalil, Bello Shehu Bello, Amina Imam AbubakararXiv 2024 · arXiv

TL;DR

This paper introduces a novel, manually annotated dataset of 4,320 tweets to analyze COVID-19 vaccine sentiment within Nigerian cyberspace. By benchmarking classical machine learning models against fine-tuned large language models (LLMs), the authors demonstrate that transformer-based models like XLM-RoBERTa excel at capturing nuanced public sentiment, even when dealing with code-mixed regional languages.

What Problem Does This Paper Address?

Public sentiment is a critical driver of vaccine acceptance and the overall success of public health initiatives. While social media sentiment analysis is widely used to gauge public opinion, most existing datasets and models focus heavily on Western contexts and standard English.

There is a significant gap in high-quality, localized datasets capturing public discourse in African contexts. In Nigeria, online discussions frequently feature Nigerian English code-mixed with local languages and regional slang. Standard sentiment analysis tools fail to capture these nuances, making it difficult for public health officials to accurately monitor vaccine hesitancy or misinformation within the region.

Methods

The authors developed a localized pipeline consisting of data collection, manual annotation, and model benchmarking:

  • Data Collection: Tweets were scraped using specific keywords (e.g., vaccin, pfizer, moderna, microchips, infertility*) and filtered strictly by Nigerian location IDs.
  • Manual Annotation: A rigorous three-annotator pipeline classified tweets into Positive, Negative, Neutral, or Indeterminate. Disagreements were resolved using a strict majority-voting and adjudication protocol.
  • Modeling: The authors benchmarked two distinct paradigms:
  • Classical Machine Learning: Support Vector Machines (SVM), Logistic Regression, Naive Bayes, K-Nearest Neighbours (KNN), and Decision Trees.
  • Fine-tuned LLMs: BERT-base, Afro-XLMR (a language model adapted for African languages), and XLM-RoBERTa-base.

Datasets

The primary contribution is a newly curated, novel dataset consisting of 4,320 preprocessed and manually annotated tweets from Nigerian cyberspace, spanning from November 1, 2020, to June 30, 2022. The dataset captures real-world public discourse surrounding major vaccine brands (AstraZeneca, Pfizer, Moderna, Johnson & Johnson) alongside prevalent conspiracy theories.

Key Insights and Ideas

  • Dominant Neutrality: Over half of the analyzed tweets (54.06%) were neutral. This indicates that a large portion of the digital population used social media to share general updates or news rather than expressing strong polarization.
  • Context is King: Fine-tuned LLMs consistently outperformed classical machine learning models. Because LLMs leverage transformer architectures trained to grasp contextual relationships, they are far better suited to handle the linguistic complexities of Nigerian code-mixed English.
  • Cross-Lingual Transferability: Interestingly, the authors found that fine-tuning a massively multilingual model (like XLM-RoBERTa) yields highly competitive results on localized datasets, even if the model was not explicitly pre-trained on the specific regional code-mixed dialects.
  • Linguistic Divergence: Word cloud analysis revealed distinct emotional markers. Positive tweets heavily relied on words like safe, free, protect, and help, whereas negative sentiment was dominated by risk-associated terms like kill, pain, conspiracy, and infected.

Key Results

  • Sentiment Distribution: The dataset breakdown revealed 54.06% Neutral, 27.20% Positive, and 18.74% Negative sentiments.

    Sentiment distribution of tweets in the dataset.
  • Classical ML Performance: Among the traditional models, SVM and Logistic Regression achieved the highest F1-score of 0.65, though SVM showed superior overall accuracy (0.66). Naive Bayes performed the poorest with an F1-score of 0.45.

    Performance metrics of classical machine learning models.
  • LLM Performance: Fine-tuned LLMs set the performance ceiling. XLM-RoBERTa base achieved the top performance with an F1-score of 0.71 and an accuracy of 0.71. BERT base followed closely with an F1-score of 0.70, demonstrating the robust capability of transformer architectures on small, specialized datasets.

Limitations and Future Work

The authors note two primary limitations:

  1. Dataset Scale: Comprising fewer than 4,400 samples, the dataset is relatively small, which can restrict the generalization capabilities of deeply parameterized deep learning models.
  2. Label Imbalance: The heavy skew toward neutral tweets introduces a class imbalance that can bias model predictions.

Future work will focus on expanding the dataset size, exploring more advanced text preprocessing techniques, and conducting deeper hyperparameter optimization to squeeze more performance out of the fine-tuned language models.

Summary updated

Abstract

Numerous successes have been achieved in combating the COVID-19 pandemic, initially using various precautionary measures like lockdowns, social distancing, and the use of face masks. More recently, various vaccinations have been developed to aid in the prevention or reduction of the severity of the COVID-19 infection. Despite the effectiveness of the precautionary measures and the vaccines, there are several controversies that are massively shared on social media platforms like Twitter. In this paper, we explore the use of state-of-the-art transformer-based language models to study people's acceptance of vaccines in Nigeria. We developed a novel dataset by crawling multi-lingual tweets using relevant hashtags and keywords. Our analysis and visualizations revealed that most tweets expressed neutral sentiments about COVID-19 vaccines, with some individuals expressing positive views, and there was no strong preference for specific vaccine types, although Moderna received slightly more positive sentiment. We also found out that fine-tuning a pre-trained LLM with an appropriate dataset can yield competitive results, even if the LLM was not initially pre-trained on the specific language of that dataset.
BibTeX
@misc{ahmad2024analyzing,
  author = {Ibrahim Said Ahmad and Lukman Jibril Aliyu and Abubakar Auwal Khalid and Saminu Muhammad Aliyu and Shamsuddeen Hassan Muhammad and Idris Abdulmumin and Bala Mairiga Abduljalil and Bello Shehu Bello and Amina Imam Abubakar},
  title = {{Analyzing COVID-19 Vaccination Sentiments in Nigerian Cyberspace: Insights from a Manually Annotated Twitter Dataset}},
  year = {2024},
  eprint = {2401.13133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.13133},
}
Figure