DuluthNLP at SemEval-2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset

Samuel Akrah, Ted PedersenACL 2023

TL;DR

This paper tackles the challenge of sentiment analysis for low-resource African languages, specifically Twi, using Twitter data. The authors introduce TwiBERT, a novel pre-trained model for Twi, and evaluate various multilingual models, finding that TwiBERT and mBERT perform best, achieving a respectable F1 score of 64.29% in the SemEval-2023 competition.

What Problem Does This Paper Address?

Sentiment analysis, a well-established NLP task for high-resource languages like English, faces significant hurdles when applied to low-resource languages, particularly those spoken in Africa. The continent boasts over 2,000 languages, yet suffers from a severe lack of readily available NLP resources and training data. This scarcity hinders the development of effective NLP tools for these languages. SemEval-2023 Task 12, AfriSenti-SemEval, aims to bridge this gap by focusing on sentiment analysis for 14 African languages using Twitter data. This paper details the DuluthNLP team's participation in this task, specifically focusing on the Twi language.

Methods

The researchers experimented with several approaches to sentiment classification for Twi. They evaluated four existing multilingual large language models (LLMs): mBERT, RoBERTa, XLM-R, and AfroXLMR-large. Crucially, they also introduced TwiBERT, a new monolingual model pre-trained from scratch on a Twi corpus (specifically, the Asanti Twi Bible and crowdsourced data). All models were fine-tuned on the Twi sentiment dataset provided by the SemEval task. For comparison, they also established baselines using a Naive Bayes classifier and a random guessing model. The fine-tuning process involved standard techniques like using the AdamW optimizer with a linear scheduler and a learning rate of 4e-5 for 16 epochs.

Datasets

The primary dataset used in this work is the AfriSenti Corpus, curated for SemEval-2023 Task 12. This corpus contains over 110,000 annotated tweets from 14 African languages, including Twi, Amharic, Hausa, Igbo, and others. Tweets were labeled with three sentiment classes (negative, positive, neutral) based on a majority vote from three annotators. The Twi portion of this dataset was used for training, development, and official evaluation. The authors also mention that the TwiBERT model was pre-trained on the Asanti Twi Bible and crowdsourced data, which is a novel contribution to the Twi language NLP landscape.

Key Insights and Ideas

  • The unique challenges of low-resource language data: The paper highlights significant issues with the Twi dataset, including the omission or replacement of proper Twi characters (due to keyboard limitations), non-standardized writing styles, and potential mislabeling. These factors make it difficult for models to learn effectively.
  • The value of monolingual pre-training: The introduction of TwiBERT, a model pre-trained specifically on Twi, demonstrates the potential benefits of language-specific pre-training, even with a relatively small dataset.
  • Multilingual models are not a silver bullet: While multilingual models like mBERT show promise, their performance can be limited by the lack of specific Twi data in their pre-training. The paper suggests that even with multilingual models, fine-tuning on the target language's data is crucial.
  • Concerns about evaluation metrics and expertise: The authors raise important ethical considerations regarding SemEval's leaderboard-focused evaluation. They argue that this approach can favor teams with significant computational resources over those with genuine linguistic expertise, potentially devaluing the contributions of native speakers and linguists.

Key Results

  • The DuluthNLP team's mBERT model achieved an official score of 64.29% on the SemEval-2023 Task 12 (Twi), ranking 14th out of 30 participating systems.
  • The TwiBERT model achieved an F1 score of 65.70% on the development data, outperforming other models in their internal evaluation.
  • On the development data, TwiBERT and mBERT performed best, with TwiBERT achieving 65.70% F1 and mBERT achieving 65.3% F1.
  • Baseline models showed significantly lower performance: Naive Bayes achieved 61% F1, and random guessing achieved 33.3% F1.
  • The top-ranked system for Track 10 achieved an F1 score of 68.28%.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size and Scope: The TwiBERT model was trained on a relatively small dataset (approximately 5MB), which may limit its ability to learn intricate contextual embeddings and generalize effectively.
  • Religious Bias: The focus on the Twi Bible for pre-training TwiBERT could potentially introduce a strong religious bias into the model's output.
  • Data Quality Issues: The presence of non-standard writing, character replacements, and potential mislabeling in the Twi dataset significantly impacts model performance.

Future work could involve:

  • Expanding the Twi pre-training dataset to include more diverse text sources.
  • Investigating methods to mitigate the potential religious bias in TwiBERT.
  • Developing more robust data cleaning and pre-processing techniques to address the challenges of low-resource language data.
  • Exploring more sophisticated evaluation methodologies that incorporate linguistic expertise.

Summary updated

BibTeX
@inproceedings{akrah2023duluthnlp,
  author = {Samuel Akrah and Ted Pedersen},
  title = {{DuluthNLP at SemEval-2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure