NLP-LISAC at SemEval-2023 Task 12: Sentiment Analysis for Tweets expressed in African languages via Transformer-based Models

Abdessamad Benlahbib, Achraf BoumhidiACL 2023

TL;DR

This paper presents transformer-based models for sentiment analysis on tweets in 14 African languages, addressing the critical need for NLP tools in under-resourced linguistic communities. The authors achieved strong results, including first place for Algerian Arabic, by fine-tuning various pre-trained models on the newly introduced AfriSenti-SemEval dataset.

What Problem Does This Paper Address?

Sentiment analysis, a cornerstone of Natural Language Processing (NLP), aims to understand the emotional tone (positive, negative, neutral) expressed in text. While significant progress has been made for high-resource languages like English, African languages remain largely underserved. This gap is due to several factors: a scarcity of annotated data, complex grammatical structures, and diverse dialects. This paper tackles this challenge by developing and evaluating transformer-based models for sentiment analysis on Twitter data from 14 African languages, contributing to the broader goal of making NLP more inclusive.

Methods

The core methodology employed by the authors is fine-tuning pre-trained transformer models. They leverage a variety of established and specialized models, adapting them to the specific sentiment analysis task for each African language.

  • Transformer-based Models: The foundation of their approach is the use of transformer architectures, which have revolutionized NLP due to their ability to capture long-range dependencies in text.
  • Fine-tuning: Instead of training models from scratch, which would be infeasible given the limited data, the researchers take models already trained on massive text corpora (often multilingual) and further train them on the specific sentiment analysis datasets for each African language. This process allows the models to adapt their general language understanding to the nuances of sentiment expression in the target languages.
  • Specialized Models: The paper highlights the use of models specifically pre-trained or fine-tuned for certain African languages, such as xlm-roberta-base-finetuned-hausa, DziriBERT (for Algerian Arabic), and AfriBERTa large (a multilingual model pre-trained on several African languages). These specialized models are expected to perform better due to their prior exposure to the linguistic characteristics of the target languages.

Datasets

The primary dataset used in this work is the AfriSenti-SemEval dataset, introduced for SemEval-2023 Task 12.

  • AfriSenti-SemEval Dataset: This dataset comprises tweets in 14 African languages, labeled with three sentiment classes: positive, negative, and neutral. The tweets were annotated by three annotators, following specific guidelines.
  • Task A (Monolingual Sentiment Classification): This sub-task utilizes training data in the specific language for classification. The paper focuses on this sub-task for most of its experiments.
  • Task C (Zero-Shot Sentiment Classification): This sub-task involves classifying tweets in Tigrinya and Oromo without any labeled training data for those languages, leveraging knowledge from other languages.

Key Insights and Ideas

  • Transformer Models are Effective for Low-Resource Languages: The success of fine-tuning pre-trained transformer models, even those not specifically trained on all target languages, demonstrates their power in bridging the gap for low-resource languages.
  • Specialized Models Offer an Edge: Models pre-trained or fine-tuned on specific African languages (e.g., DziriBERT for Algerian Arabic) often achieve superior performance, highlighting the importance of language-specific resources.
  • Multilingual Models Show Promise but Have Limitations: While multilingual models like mDeBERTa V3 base perform well across several languages, they sometimes underperform compared to specialized models, especially when the target language has unique linguistic features.
  • Zero-Shot Performance is Challenging: The zero-shot tasks proved to be particularly difficult, with AfriBERTa large showing mixed results, indicating that transferring knowledge across vastly different languages without direct training is a significant hurdle.
  • Data Quality and Annotation are Crucial: The paper implicitly emphasizes the importance of the annotated AfriSenti-SemEval dataset, noting that even with a relatively small amount of data (~3 million sequences for Moroccan Darija), good performance can be achieved with appropriate models.

Key Results

The authors achieved competitive results across various tracks of the SemEval-2023 Task 12:

  • 1st Place in Algerian Arabic (Track 6): Their fine-tuned DziriBERT model secured the top position for Algerian Arabic sentiment classification.
  • 3rd Place in Moroccan Arabic/Darija (Track 7): The mDeBERTa V3 base model achieved third place for Moroccan Arabic/Darija.
  • Multilingual Performance: For the multilingual task (Track 16), their mDeBERTa V3 base model achieved an F1 score of 67.30%, ranking 17th out of 33 participants.
  • Monolingual Performance:
    • Hausa: 79.74 F1 score (Rank 14)
    • Igbo: 79.66 F1 score (Rank 10)
    • Yoruba: 75.42 F1 score (Rank 14)
    • Algerian Arabic: 74.20 F1 score (Rank 1)
    • Moroccan Darija: 62.11 F1 score (Rank 3)
  • Zero-Shot Performance: AfriBERTa large showed poor results on Tigrinya and average results on Oromo.
  • XLM-ROBERTa (base) Performance: This model yielded average results for Hausa, Yoruba, and Igbo, but poor results for Swahili and Kinyarwanda.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Limited Data for Some Languages: Despite the creation of the AfriSenti-SemEval dataset, the amount of annotated data for some African languages remains limited, impacting model performance.
  • Zero-Shot Challenges: The zero-shot sentiment classification tasks proved to be particularly difficult, suggesting that more advanced techniques are needed for effective cross-lingual transfer in these scenarios.
  • Future Work:
    • Incorporating External Resources: The authors plan to improve results by integrating external sentiment corpora and lexicons specifically for African languages.
  • Exploring Other Transformer Models: Further investigation into other Transformer-based models, particularly those pre-trained on African languages, is suggested.
  • Improving Zero-Shot Transfer: Developing more robust methods for zero-shot sentiment analysis across diverse African languages is a key area for future research.

Summary updated

BibTeX
@inproceedings{benlahbib2023nlp,
  author = {Abdessamad Benlahbib and Achraf Boumhidi},
  title = {{NLP-LISAC at SemEval-2023 Task 12: Sentiment Analysis for Tweets expressed in African languages via Transformer-based Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure