Uppsala University at SemEval-2023 Task12: Zero-shot Sentiment Classification for Nigerian Pidgin Tweets

Annika Kniele, Meriem BeloucifACL 2023

TL;DR

This paper tackles the challenge of sentiment classification for Nigerian Pidgin, a low-resource language, by exploring zero-shot learning approaches. They found that while fine-tuning on Nigerian Pidgin itself yields the best results, using English as a zero-shot source language offers a promising alternative, outperforming other African languages like Igbo and Hausa due to linguistic similarities.

What Problem Does This Paper Address?

Sentiment classification, a common task in Natural Language Processing (NLP), is well-established for high-resource languages like English. However, for many languages with fewer available digital resources (low-resource languages), building effective sentiment classifiers is a significant hurdle. This paper focuses on Nigerian Pidgin, an English-based Creole language spoken by millions, which, despite its large speaker base, is considered low-resource in terms of NLP data. The core problem is how to achieve good sentiment classification performance for Nigerian Pidgin when labeled data is scarce, exploring the potential of leveraging knowledge from high-resource languages through zero-shot learning.

Methods

The researchers investigate the effectiveness of different pre-trained language models and fine-tuning strategies for zero-shot sentiment classification on Nigerian Pidgin tweets. They compare two main types of pre-trained models:

  • Monolingual BERT (bert-base-uncased): Pre-trained exclusively on English data.
  • Multilingual BERT (bert-base-multilingual-uncased): Pre-trained on 102 languages, excluding Nigerian Pidgin, Igbo, and Hausa.

They set up several experimental scenarios:

  • Zero-shot: Models pre-trained on English or Igbo/Hausa are fine-tuned on their respective languages and then tested on Nigerian Pidgin.
  • Monolingual Fine-tuning: Models are fine-tuned directly on Nigerian Pidgin data (both mono- and multilingual models). These serve as baselines.
  • Multilingual Fine-tuning: Models are fine-tuned on a combined dataset of Igbo and Hausa, then tested on Nigerian Pidgin.

Preprocessing steps include removing duplicates, URLs, mentions, extra whitespace, punctuation, emojis, and specific Unicode symbols, as well as lowercasing all text.

Datasets

The study utilizes datasets from two main sources:

  • Nigerian Pidgin, Igbo, and Hausa: Data provided by Muhammad et al. (2023a) for the AfriSenti-SemEval shared task. Each tweet is labeled as "positive," "negative," or "neutral."
  • English: The development set from SemEval-2017 Task 4 (Rosenthal et al., 2017), split into training and development sets (80:20). This dataset also contains "positive," "negative," and "neutral" labels.

The paper notes that the Nigerian Pidgin dataset is particularly imbalanced, with a very small number of neutral tweets, which can pose challenges for model training.

Key Insights and Ideas

  • Linguistic Proximity Matters: The study strongly suggests that the lexical and structural closeness between English and Nigerian Pidgin is a key factor. This similarity allows English-based models to transfer knowledge more effectively in a zero-shot setting compared to languages like Igbo and Hausa, which are geographically close but linguistically more distant.
  • Monolingual Fine-tuning is King: Unsurprisingly, fine-tuning a model directly on Nigerian Pidgin data yields the highest performance, highlighting the value of language-specific data.
  • Zero-Shot English is a Viable Alternative: When direct fine-tuning isn't possible, using English as the source language for zero-shot learning provides a significantly better performance than using Igbo and Hausa. This is a crucial finding for low-resource scenarios where direct data is limited.
  • Imbalanced Data is a Major Hurdle: The severe imbalance in the Nigerian Pidgin dataset, particularly the scarcity of neutral tweets, leads to models performing very poorly on neutral sentiment classification. This underscores the need for balanced datasets or advanced techniques to handle class imbalance.

Key Results

  • Best Performance: The best test accuracy was achieved by the mono-pcm setup (fine-tuned and tested on Nigerian Pidgin) at 68.9%. The multi-pcm setup also performed well at 66.8%.
  • Zero-Shot English Performance: In zero-shot scenarios, fine-tuning on English yielded test accuracies of 32.5% (mono-en) and 35.1% (multi-en). This was notably better than fine-tuning on Igbo and Hausa (32.3% mono-igha, 31.6% multi-igha).
  • F1-Scores: Similar trends were observed with F1-scores, with mono-pcm achieving 65.0% and multi-pcm at 62.5%. Zero-shot English achieved F1-scores of 36.9% (mono-en) and 39.6% (multi-en).
  • Recall Analysis: Models fine-tuned on Nigerian Pidgin recalled over 80% of negative tweets but struggled significantly with neutral tweets (near 0% recall). English fine-tuning showed better recall for neutral tweets (around 80%) but lower recall for negative tweets.

Limitations and Future Work

  • Data Imbalance: The authors acknowledge the significant class imbalance in the Nigerian Pidgin dataset, particularly the low number of neutral tweets, which severely impacts the models' ability to classify neutral sentiments.
  • Random Seed Variability: The paper notes that the observed differences in performance between English and Igbo/Hausa zero-shot scenarios might be influenced by the specific random seeds used, suggesting high variability in results.
  • Future Directions:
    • Investigate techniques to alleviate data imbalance, such as undersampling or data augmentation.
  • Explore the effect of using both Nigerian Pidgin and English data for fine-tuning.
  • Further research into balancing datasets or using synthetic data.

Summary updated

BibTeX
@inproceedings{kniele2023uppsala,
  author = {Annika Kniele and Meriem Beloucif},
  title = {{Uppsala University at SemEval-2023 Task12: Zero-shot Sentiment Classification for Nigerian Pidgin Tweets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure