TL;DR
This paper tackles the challenge of sentiment analysis for low-resource African languages by applying traditional machine learning models and the multilingual BERT (mBERT) model to a Twitter dataset. The authors highlight the performance differences between these approaches, particularly when dealing with imbalanced datasets, and offer insights into the effectiveness of different oversampling techniques.
What Problem Does This Paper Address?
Sentiment analysis, a crucial application of Natural Language Processing (NLP), has seen significant advancements for high-resource languages. However, a substantial gap exists for low-resource languages, particularly those spoken in Africa. Many of these languages lack sufficient curated datasets, hindering the development of effective AI applications like sentiment analysis. This paper addresses this gap by participating in SemEval-2023 Task 12, which focuses on sentiment analysis for 17 African languages using Twitter data. The authors aim to evaluate the performance of various machine learning models on three specific African languages: Hausa, Yoruba, and Swahili, which are often underrepresented in NLP research.
Methods
The researchers employed a combination of traditional machine learning algorithms and a state-of-the-art transformer model. The traditional methods included:
- Logistic Regression: A fundamental supervised classification algorithm.
- Random Forest: An ensemble technique that combines multiple decision trees for improved accuracy and robustness.
- Support Vector Machine (SVM): A powerful algorithm for classification, particularly effective in high-dimensional spaces.
These traditional models were integrated into pipelines that incorporated data preprocessing and oversampling techniques. Specifically, they utilized:
- TF-IDF (Term Frequency-Inverse Document Frequency): A common technique for converting text into numerical representations, tuned with unigrams and bigrams.
- Random OverSampler (ROS): A method that addresses class imbalance by randomly duplicating instances from the minority class.
- SMOTE (Synthetic Minority Over-sampling Technique): An algorithm that generates synthetic data points for the minority class to balance the dataset.
For a more advanced approach, they also implemented:
- mBERT (Multilingual BERT): A pre-trained transformer model capable of understanding and processing text in 104 different languages. This model was used with the
ktrainAPI for ease of implementation.
The performance of these models was evaluated using weighted-average and macro-average F1 scores.
Datasets
The primary dataset used in this study was provided by the SemEval-2023 Task 12, specifically the "AfriSenti" dataset. This dataset consists of tweets in various African languages. The authors utilized the following splits:
- Training data: 'language-train.tsv' files for each language.
- Validation data: 'language-dev.tsv' files for model tuning.
- Test data: 'language-test.tsv' files for final evaluation and prediction.
The paper provides the sizes of these splits for Hausa, Yoruba, and Swahili:
- Hausa: Train (14172), Dev (2677), Test (5303)
- Yoruba: Train (8552), Dev (2090), Test (4515)
- Swahili: Train (1810), Dev (453), Test (748)
Key Insights and Ideas
- mBERT's potential for balanced data: The authors observed that when datasets are balanced, deep learning models like mBERT tend to be more efficient.
- Traditional methods shine with imbalance: For imbalanced datasets, traditional classification methods combined with oversampling techniques (like SMOTE) can yield better performance.
- Oversampling techniques matter: The choice of oversampling technique (ROS vs. SMOTE) had a noticeable impact on the performance of traditional classifiers, with SMOTE often leading to better results.
- Language-specific performance: The effectiveness of models varied across the three languages, underscoring the unique linguistic characteristics and potential data imbalances within each.
Key Results
The paper reports the best F1 scores achieved for each language using different models and techniques. Notably:
- Hausa: mBERT achieved the highest weighted-average F1 score of 76.53 and a macro-average F1 score of 76.55.
- Yoruba: SVC (SMOTE) performed best with a weighted-average F1 score of 74.83 and a macro-average F1 score of 73.15.
- Swahili: Logistic Regression (SMOTE) yielded the best results with a weighted-average F1 score of 57.79 and a macro-average F1 score of 48.59.
It's important to note that only the mBERT results were submitted for the competition, leading to rankings of 24th for Hausa, 31st for Yoruba, and 29th for Swahili in subtask-1.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited data preprocessing: They were unable to implement extensive data scraping or crawling techniques, which they believe impacted overall accuracy.
- Lack of AfriBERT and other specialized models: The implementation of models like AfriBERT and other pre-trained models specifically for African languages was not feasible within the scope of their submission.
- Limited tokenizer exploration: They suggest that trying various tokenizers could lead to better preprocessing.
Future work proposed by the authors includes:
- Training models using datasets built through data scraping and crawling.
- Experimenting with different tokenizers for improved preprocessing.
- Implementing and evaluating specialized models like AfriBERT for African languages.
Summary updated