TL;DR
This paper presents an ensemble learning approach for sentiment analysis on African languages, combining transformer-based models with traditional statistical methods and emoticon features. The system achieved a respectable 7th place in three languages, highlighting the challenges and opportunities in low-resource language NLP.
What Problem Does This Paper Address?
Sentiment analysis, a crucial task in understanding public opinion and user feedback, has seen significant advancements for high-resource languages. However, there's a notable gap in research and available resources for African languages. The SemEval-2023 Task 12: AfriSenti-SemEval aimed to address this by providing a benchmark for sentiment classification across 14 diverse African languages, many of which are considered low-resource. This paper tackles the challenge of building effective sentiment analysis models for these languages, where data scarcity and linguistic diversity pose significant hurdles.
Methods
The core of the proposed system is an XGBoost-based ensemble model. This ensemble leverages the predictive power of several individual models:
- Statistical Models: Logistic Regression (LR), Support Vector Machines (SVM) with a linear kernel, and Random Forests (RF). These models were trained using TF-IDF features.
- Transformer-based Models:
- AfriBERTa: An XLM-RoBERTa model pre-trained on 11 African languages.
- AfroXLMR: Another XLM-RoBERTa variant trained on 17 African languages.
- AfriBERTa-NaijaSenti: A fine-tuned version of AfriBERTa-large, specifically trained on the NaijaSenti dataset.
- Emoticon Features: The frequency of commonly occurring emoticons (present in at least 10% of training samples) was also incorporated as a feature.
The system was trained on the provided training sets for each language, and the development set was used for evaluation during the competition. The final ensemble model takes the predicted probabilities from these individual classifiers and emoticon features as input to predict the sentiment (positive, neutral, or negative).
Datasets
The primary dataset used is the AfriSenti dataset, provided by the organizers for SemEval-2023 Task 12. This dataset consists of manually annotated tweets in 12 African languages. The task also included a multilingual dataset created by combining these individual language datasets. The paper notes that some languages exhibit significant class imbalance, with less than 2% of samples labeled as neutral in Nigerian Pidgin, and less than 20% representation for one class in Xitsonga, Twi, and Swahili.
Key Insights and Ideas
- Ensembling for Robustness: The paper strongly advocates for ensemble learning, particularly combining diverse models (statistical and transformer-based) to achieve more robust sentiment predictions.
- Transformer Models' Dominance: While statistical models contribute, the results suggest that transformer-based models, especially AfriBERTa and AfroXLMR, are crucial for achieving competitive performance.
- Emoticon Importance (and Limitations): Emoticons are recognized as important indicators of sentiment. However, their effectiveness as features can be limited by their sparsity in the dataset, potentially leading to lower recall.
- Class Imbalance Challenges: The authors observed that significant class imbalance in the training data negatively impacts model performance, particularly for the neutral class.
- Language Similarities: The study found interesting correlations in performance between certain language pairs (e.g., Hausa, Igbo, Yoruba; Kinyarwanda-Swahili; Nigerian Pidgin-Amharic), suggesting potential typological or resource-related commonalities.
Key Results
The system achieved a best rank of 7th in the Igbo, Twi, and Yoruba tracks. It also ranked between 12th and 15th in the multilingual, Xitsonga, and Hausa tracks. The paper provides detailed weighted F1 scores for individual models and the ensemble across languages in Table 3. For instance, the ensemble achieved a weighted F1 score of 80.87% for Igbo and 78.44% for Yoruba.
An ablation study revealed that ensembling only the transformer-based models often outperformed other configurations. Notably, for Darija, the ensemble of statistical models performed better, possibly due to its mixed script nature.
Limitations and Future Work
- Class Imbalance: The authors acknowledge that class imbalance in the training data negatively affected their system's performance compared to others.
- Emoticon Feature Sparsity: The effectiveness of emoticon features was limited by their sparsity, leading to situations where their inclusion did not improve results.
- Code-Mixing: While the dataset includes code-mixed tweets (especially with English), the attempts to handle this by replacing emoticons with text did not yield improvements.
- Future Work: The authors suggest further investigation into combining datasets from related languages (like Arabic variants) to create more robust models. They also propose exploring linguistic and typological features of correlated language pairs to understand performance similarities. Additionally, they aim to investigate the impact of different ensemble configurations and feature sets more deeply.
Summary updated