TL;DR
This paper presents UMUTeam's approach to the SemEval-2023 AfriSenti task, focusing on sentiment analysis for low-resource African languages. They achieved strong results by ensembling predictions from various Large Language Models (LLMs) with linguistic features, particularly excelling in languages with less training data.
What Problem Does This Paper Address?
Sentiment analysis, a crucial task in understanding public opinion and user feedback, has historically been dominated by well-resourced languages like English. However, a significant portion of the world's population speaks languages for which extensive labeled data and pre-trained models are scarce. The SemEval-2023 AfriSenti shared task aimed to bridge this gap by focusing on sentiment analysis for 16 African languages, many of which have limited digital resources. This presents a significant challenge: how to build effective sentiment analysis systems for languages with insufficient training data, where standard NLP techniques might falter. The task was divided into two subtasks: monolingual sentiment classification (treating each language separately) and multilingual sentiment classification (merging all languages).
Methods
The UMUTeam's strategy revolved around an ensemble learning approach. They combined two main types of features:
- Language-Independent Linguistic Features (LFs): These were extracted using the UMU-TextStats tool, focusing on stylometric and morphosyntactic features that are not tied to a specific language. This was achieved by leveraging Named Entity Recognition (NER) and Part-of-Speech (PoS) models from Stanza.
- Large Language Models (LLMs): They experimented with a diverse set of LLMs, including general-purpose multilingual models (BERT, XLM, DeBERTa), a dialect-specific model (DziriBERT for Algerian Arabic), and an Africa-focused multilingual model (AfriBERTa). They also included the English BERT due to a notable presence of English documents in the dataset.
The core of their method involved fine-tuning these LLMs and then ensembling their predictions. They explored three ensemble strategies:
- Highest Probability: The final prediction is based on the model that outputs the highest probability for a given sentiment.
- Average Probabilities: Probabilities from all LLMs for each sentiment are averaged, and the highest average determines the final prediction.
- Hard-Voting: The most frequent prediction across all models is chosen.
Interestingly, for the monolingual subtask, they trained all languages together, even those with limited data, to potentially boost their performance by leveraging the larger combined dataset.
Datasets
The primary dataset used was the AfriSenti 2023 shared task dataset, comprising 107,549 documents annotated for positive, negative, or neutral sentiment across 14 African languages. The authors created a custom 80-20 train-validation split from the provided training data for their internal evaluations. They also noted that some documents were misidentified by the task organizers and used fastText for language identification to correct these.
Key Insights and Ideas
- Ensembling LLMs is Powerful: Combining predictions from multiple LLMs, even those with varying strengths, consistently outperformed individual models.
- Low-Resource Languages Benefit from Shared Training: Training all languages together for the monolingual subtask seemed to benefit languages with fewer instances, leading to better results for them.
- Language-Independent Features Complement LLMs: While LLMs captured most of the sentiment signal, the linguistic features provided a useful complementary signal, especially when ensembled.
- Neutral Class is a Common Error Source: Confusion matrices revealed that misclassifications often occurred between the neutral class and the positive/negative classes, highlighting the difficulty in distinguishing neutral sentiment.
- Model Performance Varies by Language: The authors observed that their performance varied significantly across different languages, suggesting that the degree of difficulty and the effectiveness of models are highly language-dependent.
Key Results
- Best Overall Performance: The ensemble of LLMs and LFs using the average probabilities strategy achieved the best weighted F1-score of 74.994% on their custom validation split for subtask 1.
- Strongest Individual Language Performance: UMUTeam achieved their best official ranking on Xitsonga (ts) with a weighted F1-score of 54.89% (7th out of 31 participants). They also performed well on Algerian Arabic (dz) (68.52%, 13th/30), Swahili (sw) (60.52%, 14th/30), and Twi (twi) (71.14%, 12th/30).
- Multilingual Task Performance: For the multilingual subtask (track 16), they achieved a weighted F1-score of 65.47%, ranking 23rd out of 33 participants.
- LLM Performance: Individual LLMs achieved weighted F1-scores ranging from 67.307% (BERT) to 72.203% (AFROXLMR) on their custom validation split.
Limitations and Future Work
- AfroXLMR-base Exclusion: The AfroXLMR-base model was not included in the official submission due to its late suggestion during the revision process, but its results were analyzed separately.
- Data Augmentation: The authors suggest incorporating data augmentation techniques to further support underrepresented languages.
- Individual Track Training: They propose training models for each track individually in the future, rather than combining all languages, to better understand language-specific nuances and cultural differences.
- Performance Discrepancies: The authors acknowledge that their results were more limited on tracks with a larger number of documents, where other teams achieved more competitive results. This suggests a need for more robust models or larger datasets for high-resource tracks.
Summary updated