TL;DR
This paper presents a system for sentiment analysis on African languages, tackling the challenge of limited data and out-of-distribution generalization. By combining and further pre-training existing language models like AfroXLM and MARBERT, and exploring advanced training objectives, the authors achieved competitive results on the AfriSenti SemEval-2023 task, particularly excelling in monolingual sentiment classification.
What Problem Does This Paper Address?
The digital world is increasingly multilingual, yet Natural Language Processing (NLP) tools often lag behind for low-resource languages, particularly those spoken in Africa. This gap is starkly evident in sentiment analysis, where data scarcity and the diversity of African languages and dialects pose significant hurdles. Existing NLP models, primarily developed for high-resource languages, struggle to generalize to these underrepresented linguistic communities. The AfriSenti SemEval-2023 task specifically aims to address this by focusing on sentiment analysis for 14 African languages and dialects, encompassing monolingual, multilingual, and zero-shot scenarios, pushing the boundaries of out-of-distribution generalization.
Methods
The researchers employed a multi-pronged approach to enhance sentiment analysis performance for African languages:
- Tweet Encoders: They experimented with two powerful pre-trained language models (PLMs):
- MARBERT: A transformer-based encoder specifically pre-trained on 1 billion Arabic tweets, covering both modern standard and dialectal Arabic.
- AfroXLM: A multilingual XLM-R PLM adapted using unlabeled text from 17 African languages and three high-resource languages (Arabic, English, French).
- DUO: A novel combination of MARBERT and AfroXLM, where MARBERT's embeddings were projected to match AfroXLM's dimensionality and then combined using a residual layer.
- Further Pre-training: To better adapt AfroXLM to the nuances of tweet data and African languages, they created a custom pre-training dataset by augmenting the AfriSenti training data with existing African text corpora. This adapted model was then further pre-trained using the "whole word masking" objective.
- Training Objectives: Beyond standard cross-entropy loss, they explored several auxiliary losses to improve generalization:
- SoftTriple Loss: For class-wise text embedding alignment, aiming to pull embeddings of the same sentiment closer and push different sentiments apart.
- Correlation Alignment (CORAL): A domain generalization technique to align the distributions of features across different domains.
- Regularized Mixup (RegMixup): A data augmentation technique that generates new training samples by interpolating existing ones and their labels.
- For models combining multiple losses, they used an automatically weighted multi-task loss to balance their importance.
Datasets
The primary dataset used in this work is the AfriSenti SemEval-2023 Task 12 dataset. This dataset comprises tweets in 14 African languages and dialects, labeled with negative, neutral, or positive sentiment polarities. It supports three tasks:
- Task A: Monolingual Sentiment Classification: 12 tracks covering various languages.
- Task B: Multilingual Sentiment Classification: A combined dataset of Task A languages.
- Task C: Zero-Shot Sentiment Classification: Focused on Tigrinya and Oromo, leveraging Task A data.
In addition to the AfriSenti dataset, the authors created a custom pre-training dataset by combining:
- The AfriSenti training data (duplicated).
- WebCrawl African multilingual parallel corpora.
- The lafand-mt dataset.
- The African News Corpus.
- The Maghrebi partition of the IADD dataset.
Key Insights and Ideas
- Combining Models Matters: The DUO model, which combines MARBERT and AfroXLM, consistently outperformed using either model individually, highlighting the benefit of ensemble-like approaches even at the encoder level.
- Further Pre-training is Beneficial (Mostly): Adapting AfroXLM through further pre-training (AfroXLM_wwm) generally improved performance, especially on monolingual tasks. However, this adaptation had a negative impact on zero-shot performance for Tigrinya and Oromo, suggesting that the pre-training data might not adequately cover these specific languages.
- Out-of-Distribution Generalization Techniques Pay Off: Employing metric learning (SoftTriple loss) and domain generalization techniques (CORAL, RegMixup) proved crucial for improving performance, particularly on the multilingual and zero-shot tasks.
- Dialectal Nuances are Important: MARBERT, pre-trained on Arabic dialects, showed superior performance on Moroccan and Algerian dialects compared to AfroXLM, underscoring the value of domain-specific pre-training.
Key Results
The system achieved promising results across the AfriSenti tasks:
- Task A (Monolingual): The DUO_wwm_st model achieved the best average performance. MARBERT outperformed AfroXLM on Moroccan and Algerian dialects, while AfroXLM_wwm showed improvements over the original AfroXLM on most tracks.
- Task B (Multilingual): The DUO_wwm_st_coral_mix model, combining the adapted AfroXLM and MARBERT with metric learning and out-of-distribution generalization losses, yielded the best results.
- Task C (Zero-Shot): While adaptive pre-training negatively impacted performance, the DUO_wwm model performed best on the Oromo track. For Tigrinya, the DUO_wwm_st_coral_mix model achieved the best results.
- Official Submissions: The team submitted DUO_wwm for Task A and DUO_wwm_st_coral_mix for Tasks B and C. They ranked among the top ten systems on seven tracks of Task A, and were ranked 6th on Task B and 6th and 4th on the zero-shot Tigrinya and Oromo tracks of Task C, respectively.
Limitations and Future Work
- Impact of Pre-training Data: The authors acknowledge that the negative impact of adaptive pre-training on Task C might be due to the limited coverage of Tigrinya and Oromo in their custom pre-training dataset. Future work could involve curating more targeted pre-training data for these specific languages.
- Generalization to Unseen Languages: While the system shows promise, further research is needed to improve generalization to entirely unseen African languages and dialects.
- Exploring Other Architectures and Objectives: The paper focuses on specific PLMs and training objectives. Future work could explore other state-of-the-art models and more advanced domain generalization techniques.
- Data Augmentation: Investigating more sophisticated data augmentation strategies tailored for low-resource languages could further enhance robustness.
Summary updated