Seals_Lab at SemEval-2023 Task 12: Sentiment Analysis for Low-resource African Languages, Hausa and Igbo

Nilanjana Raychawdhary, Amit Das, Gerry Dozier, Cheryl D. SealsACL 2023

TL;DR

This paper tackles the challenge of sentiment analysis for low-resource African languages, specifically Hausa and Igbo, by fine-tuning the pre-trained multilingual language model AfriBERTa large on a newly compiled Twitter dataset. The research demonstrates that even with limited data, deep learning models can achieve competitive performance, achieving top-10 rankings in the SemEval-2023 Task 12.

What Problem Does This Paper Address?

Sentiment analysis (SA), the task of determining the emotional tone of text, is a well-established NLP problem. However, most research and available resources focus on high-resource languages like English. Low-resource languages, particularly those spoken in Africa, suffer from a significant lack of annotated data, making it difficult to apply existing NLP techniques or develop effective SA systems. This scarcity hinders the development of AI applications that can understand and process the nuances of communication in these languages, which represent a substantial portion of global linguistic diversity. The paper specifically addresses this gap by focusing on Hausa and Igbo, two widely spoken Nigerian languages, and the challenges posed by code-mixed tweets within their datasets.

Methods

The core of this research lies in leveraging and adapting existing deep learning models for low-resource SA. The authors employed a transformer-based deep learning model, specifically AfriBERTa large, which is a multilingual language model pre-trained on 11 African languages. The methodology involved:

  • Fine-tuning: The AfriBERTa large model was fine-tuned on the AfriSenti-SemEval Shared Task 12 Twitter datasets for Hausa and Igbo. This process adapts the general language understanding capabilities of the pre-trained model to the specific task of sentiment classification for these languages.
  • Preprocessing: Standard text preprocessing techniques were applied to normalize the data and prepare it for model training.
  • Experimental Setup: The research followed a three-phase experimental setup: training, evaluation, and ranking. This involved using labeled training and development datasets for model tuning and then evaluating performance on unseen test datasets.
  • Model Selection: Through initial experiments with various models (XLM-ROBERTa-base, Afro-xlmr-mini, mDeBERTaV3-base, RemBERT), AfriBERTa large was identified as the most promising for this task due to its prior success in African language NLP.

Datasets

The primary dataset used in this research is the AfriSenti-SemEval Shared Task 12 Twitter datasets. This dataset was specifically compiled for the SemEval-2023 Task 12 and includes human-annotated tweets in Hausa and Igbo. The dataset is broken down into training, development, and test sets for each language.

  • Hausa: Training set (14,173 tweets), development set (2,678 tweets), and test set (5,304 tweets).
  • Igbo: Training set (10,193 tweets), development set (1,842 tweets), and test set (3,683 tweets).

The paper highlights that these datasets contain a significant number of code-mixed tweets, adding complexity to the SA task.

Key Insights and Ideas

  • Viability of Pre-trained Models for Low-Resource Languages: The research strongly suggests that pre-trained multilingual models like AfriBERTa large can be effectively adapted for sentiment analysis in low-resource languages, even with limited training data. This is a crucial finding for democratizing NLP capabilities.
  • Importance of African-Specific Models: The success of AfriBERTa large, which was pre-trained on African languages, underscores the benefit of using models tailored to the linguistic nuances of these regions, rather than solely relying on models trained on data from other language families.
  • Code-Mixing as a Challenge and Opportunity: The presence of code-mixed tweets in the dataset presents a significant challenge but also highlights the need for SA systems that can handle such linguistic phenomena, which are common in multilingual communities.
  • Competitive Performance Achievable: Despite the inherent difficulties of low-resource SA, the paper demonstrates that achieving competitive performance, even ranking within the top 10 systems in a major shared task, is possible.

Key Results

The fine-tuned AfriBERTa large model achieved impressive results on the AfriSenti-SemEval Shared Task 12:

  • Hausa: Achieved an F1 score of 80.85% on the test set.
  • Igbo: Achieved an F1 score of 80.82% on the test set.

The system consistently ranked in the top 10 for both Hausa and Igbo languages in the official evaluation, with a mean F1 score exceeding 80%. Specifically, the "Seals_Lab" team ranked ninth out of 35 teams for Hausa and eighth out of 33 teams for Igbo, based on weighted F1 scores.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Data Imbalance: The datasets may suffer from class imbalance, which can affect model performance. Future work could explore techniques to address this.
  • Topic Information Integration: The current model does not explicitly incorporate topic information, which could potentially improve sentiment classification accuracy.
  • Enhanced Twitter Representation: Further research could focus on developing more sophisticated methods for representing Twitter data, potentially capturing more nuanced linguistic features.
  • Incorporating Extra Resources: Exploring the use of additional linguistic resources and features could lead to further performance improvements.

Summary updated

BibTeX
@inproceedings{raychawdhary2023seals,
  author = {Nilanjana Raychawdhary and Amit Das and Gerry Dozier and Cheryl D. Seals},
  title = {{Seals\_Lab at SemEval-2023 Task 12: Sentiment Analysis for Low-resource African Languages, Hausa and Igbo}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure