ABCD Team at SemEval-2023 Task 12: An Ensemble Transformer-based System for African Sentiment Analysis

Dang Van Thin, Dai Ba Nguyen, Dang Ba Qui, Duong Ngoc Hao, Ngan Luu-Thuy NguyenACL 2023

TL;DR

This paper presents an ensemble system for African sentiment analysis using pre-trained transformer models and a soft voting technique. The system achieves competitive results on several low-resource African languages, demonstrating the effectiveness of ensemble methods for this challenging task.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, has seen significant advancements, particularly for high-resource languages. However, applying these techniques to low-resource languages, such as many spoken in Africa, remains a substantial challenge. This is primarily due to the scarcity of labeled data and the linguistic diversity of these languages. The SemEval-2023 Task 12 specifically aimed to address this gap by focusing on sentiment analysis for 17 African languages using Twitter data, which often exhibits code-mixing (mixing of multiple languages within a single tweet). The paper tackles the problem of building effective sentiment analysis systems for these under-resourced languages, especially in monolingual, multilingual, and zero-shot settings.

Methods

The core of the ABCD team's system is an ensemble architecture built upon pre-trained transformer-based language models. The process involves three main stages:

  1. Pre-processing: Standard steps were applied to the Twitter data, including:
  2. Word Normalization: Replacing common phrases like "URL" with "website" for consistency.
  3. Noise Removal: Eliminating punctuation and special characters that don't contribute to sentiment.
  4. Emoji Replacement: Converting emojis into sentiment labels (positive, negative, neutral) to leverage their significant presence in the dataset (up to 28.85%).
  5. Fine-tuning Language Models: Three specific pre-trained transformer models were chosen for their ability to handle African languages:
  6. AfroXLMR: A large language model specifically adapted for 17 African languages and three high-resource languages.
  7. AfriBERTa: A transformer model trained on 11 low-resource African languages.
  8. LaBSE: A language-agnostic BERT sentence embedding model supporting a wide range of languages. These models were fine-tuned on the task-specific data. The output of the last transformer layer's [CLS] token was used as the input to a linear layer with a Softmax activation to predict sentiment class probabilities.

  9. Soft Voting Ensemble: To combine the predictions of the individual fine-tuned models, a simple soft voting technique was employed. This method sums the probability outputs of each base classifier and selects the sentiment class with the highest aggregated probability as the final prediction. This approach aims to leverage the strengths of each individual model.

Datasets

The primary dataset used in this work is the AfriSenti-SemEval Shared Task 12 dataset. This dataset consists of tweets in 17 African languages, including Hausa, Yoruba, Igbo, Nigerian Pidgin, Amharic, Tigrinya, Oromo, Swahili, Algerian Arabic, Kinyarwanda, Twi, Mozambican Portuguese, and Moroccan Arabic/Darija. The tweets are labeled with three sentiment classes: positive, negative, and neutral. The dataset is characterized by code-mixing. The paper mentions using the official training set and the development set for hyper-parameter tuning. For Task C (zero-shot), they utilized Google Translate to translate test sets into Hausa.

Key Insights and Ideas

  • Ensemble Power for Low-Resource Languages: The paper strongly advocates for ensemble methods, showing that combining multiple transformer models can yield better performance than any single model, especially in low-resource scenarios.
  • Emoji Importance: Recognizing and explicitly handling emojis as sentiment indicators is crucial for Twitter data, significantly boosting model performance.
  • Pre-trained Models for African Languages: The selection of models like AfroXLMR and AfriBERTa, specifically designed or adapted for African languages, is key to achieving competitive results.
  • Soft Voting Simplicity and Effectiveness: The straightforward soft voting mechanism proved effective in aggregating predictions, highlighting that complex fusion strategies aren't always necessary.
  • Challenges in Zero-Shot and Multilingual Settings: While the ensemble performed well in monolingual tasks, the paper notes challenges in multilingual and zero-shot settings, suggesting that translation errors or inherent task complexities play a role.

Key Results

The ABCD team's ensemble system achieved competitive results in the SemEval-2023 Task 12:

  • Task A (Monolingual Sentiment Analysis): The system ranked within the top positions for several languages, including Top 3 for Hausa, Top 2 for Igbo, and Top 4 for Moroccan languages.
  • Task B (Multilingual Sentiment Analysis): The system achieved an F1-score of 69.22%, ranking 14th place.
  • Task C (Zero-Shot Sentiment Analysis): The system ranked 14th place for Track 17 (Tigrinya) and 8th place for Track 18 (Oromo).

The paper also presents development set results (Table 2) showing that the ensemble model generally outperforms individual base models across most languages, though there are exceptions where individual models perform better.

Limitations and Future Work

  • Limitations:
    • The system's performance in multilingual and zero-shot tasks was not as strong as in monolingual tasks. The authors attribute this, in part, to potential errors in the translation process used for Task C.
  • The paper notes that the ensemble approach was more effective for tracks with a larger number of samples and a balanced distribution of labels.
  • For Task B, computational resource limitations led to a fold-based approach with LaBSE, which might have impacted overall performance.
  • Future Work:
    • The authors plan to improve the system's ability to handle imbalanced datasets in some languages.
  • Data augmentation is suggested as a promising direction to further enhance the system's performance.

Summary updated

BibTeX
@inproceedings{thin2023abcd,
  author = {Dang Van Thin and Dai Ba Nguyen and Dang Ba Qui and Duong Ngoc Hao and Ngan Luu-Thuy Nguyen},
  title = {{ABCD Team at SemEval-2023 Task 12: An Ensemble Transformer-based System for African Sentiment Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure