TL;DR
This paper tackles the challenge of sentiment analysis for low-resource African languages on Twitter. The team developed an ensemble of pre-trained multilingual models, achieving a respectable 19th place in the SemEval-2023 competition, highlighting the effectiveness of ensemble methods for this under-served domain.
What Problem Does This Paper Address?
Sentiment analysis, a crucial task in understanding public opinion and user feedback, has historically been dominated by high-resource languages like English. However, a vast number of African languages remain under-represented in NLP research due to a lack of curated datasets and effective pre-trained models. This paper addresses this gap by focusing on SemEval-2023 Task 12, which aims to foster interest and develop efficient sentiment analysis models for 12 diverse African languages using Twitter data. The core challenge lies in adapting existing NLP techniques to languages with limited linguistic resources and potentially different linguistic structures.
Methods
The ISCL_WINTER team employed an ensemble approach as their primary strategy. This involves combining the predictions of multiple individual models to achieve better overall performance than any single model. Specifically, they utilized a hard voting ensemble, where the final prediction is determined by the majority vote across the constituent models.
Initially, their ensemble consisted of three pre-trained models. This was later expanded to five models. The specific models included in their final ensemble were:
Davlan/afro-xlmr-mini(Alabi et al., 2022)prajjwall/bert-tiny(Bhargava et al., 2021)clips/mfaq(Bruyn et al., 2021)finiteautomata/bertweet-base-sentiment-analysis(Pérez et al., 2021)cardiffnlp/twitter-roberta-base-sentiment-latest(Barbieri et al., 2020)
They also experimented with other approaches, including:
- Translation to English: Translating tweets into English to leverage existing English-trained models.
- Translation to Arabic: Translating tweets into Arabic and using an Arabic-specific sentiment analysis model.
- Support Vector Machines (SVM): Used as a baseline and for initial trials.
Datasets
The primary dataset used in this study is the AfriSenti dataset, which consists of tweets written in one of 12 African languages. Each tweet is labeled with one of three sentiment classes: positive, negative, or neutral. The dataset was annotated by three annotators following guidelines from Muhammad et al. (2023a). The specific languages included are Hausa, Yoruba, Igbo, Nigerian Pidgin, Amharic, Tigrinya, Oromo, Swahili, Algerian Arabic dialect, Kinyarwanda, Twi, Mozambique Portuguese, and Moroccan Arabic/Darija.
Key Insights and Ideas
- Ensemble Power: The ensemble approach proved to be the most effective strategy, significantly outperforming individual models and baseline approaches. This underscores the value of combining diverse models for low-resource language tasks.
- Translation Challenges: Translating tweets to English or Arabic did not yield the expected improvements. The authors suggest that significant data loss or meaning distortion during translation might be the cause, especially for languages with unique linguistic structures or idioms.
- Emoji Impact: Tweets containing emojis were found to be predicted with higher accuracy than those without. This suggests that emojis can provide valuable sentiment cues, even in low-resource settings.
- Arabic Script Advantage: Languages with Arabic script (like Darija and Algerian Arabic) showed improved performance when using Arabic-specific models, highlighting the benefit of leveraging language-family similarities.
- Data Scarcity is Key: The authors identify the lack of data as the most significant challenge, suggesting that increasing the dataset size and providing more detailed parameter descriptions could further improve performance.
Key Results
The ISCL_WINTER team's ensemble system achieved an F1 Score of 0.68 on the final testing set, ranking 19th out of 33 participants in the SemEval-2023 competition.
- The ensemble achieved a development stage F1 Score of 0.715 and a testing stage F1 Score of 0.68.
- For languages with Arabic script, the ensemble showed notable improvements compared to the baseline model, with Darija seeing a 19% increase in performance.
- The translation to English approach resulted in a lower F1 Score of 0.559, compared to the baseline model's 0.62.
- Stripping tweets of emojis led to a loss of approximately 2% in prediction accuracy.
Limitations and Future Work
- Data Scarcity: The most significant limitation acknowledged is the lack of sufficient data for these low-resource languages.
- Translation Issues: The current translation strategies did not yield positive results, suggesting a need for more sophisticated translation libraries or alternative approaches to handle linguistic nuances.
- Unknown Performance: The reason for the worst-performing languages (Swahili and Amharic) within the Arabic script group remains unknown.
- Future Work: The authors suggest exploring more sophisticated translation libraries, investigating lexicons and data cleaning techniques, and potentially using SVM and Random Forest models to improve accuracy. They also propose finding more productive solutions for individual languages and combining them with simple language classifiers.
Summary updated