TL;DR
This paper introduces RideKE, a novel dataset of Kenyan Twitter conversations in code-switched English, Swahili, and Sheng, for sentiment and emotion analysis. They evaluate state-of-the-art transformer models, finding XLM-R to be the most effective, and demonstrate the potential of semi-supervised learning to improve performance and reduce annotation costs in low-resource settings.
What Problem Does This Paper Address?
Social media is a rich source of public opinion, but analyzing content in low-resource languages, especially those with significant code-switching (like in Kenya), presents a major challenge. Existing Natural Language Processing (NLP) tools and datasets are heavily biased towards high-resource languages, leaving a significant gap in understanding diverse linguistic communities. Specifically, leveraging user-generated content from platforms like Twitter for sentiment and emotion detection in code-switched African languages is difficult due to data scarcity, poor quality, and linguistic variations. This paper aims to address this by creating a dedicated dataset and evaluating how well current NLP models can handle this complex linguistic landscape.
Methods
The authors employed a multi-pronged approach:
- Data Collection: Tweets related to ride-hailing services (Uber, Bolt, Little Cab) in Kenya were scraped using the
snscrapePython library. - Language Detection: The
langdetectlibrary was used to identify languages, with English (code-switched) being the focus for analysis. Sheng was manually identified. - Data Preprocessing: A rigorous pipeline was used to clean and standardize tweets, including lowercasing, trimming whitespace, expanding contractions, and tokenization.
- Annotation: Two Kenyan annotators, fluent in English, Swahili, and Sheng, manually labeled a subset of tweets for sentiment (positive, negative, neutral) and emotion (anger, fear, happy, love, sadness, surprise, empathy, frustration, neutral). Cohen's Kappa was used to measure inter-annotator agreement.
- Model Evaluation: Four state-of-the-art transformer models (DistilBERT, mBERT, XLM-RoBERTa, and AfriBERTa) were evaluated using both supervised and semi-supervised learning approaches.
- Semi-Supervised Learning (SSL): To leverage the large amount of unlabeled data, an SSL loop was implemented. Models were initially trained on labeled data, then used to predict labels for unlabeled data, and retrained on this augmented dataset. A dynamic threshold was used to select high-confidence predictions for pseudo-labeling.
Datasets
- RideKE Dataset: A newly introduced dataset comprising over 29,000 tweets from Kenyan ride-hailing services. It features code-switched content primarily in Kenyan-accented English, with a mix of Swahili and Sheng. The dataset is annotated for sentiment and emotion.
- Existing Datasets: While not explicitly detailed for training the base models, the paper references the use of large, uncompressed text corpora like BOOKCORPUS and CommonCrawl English for pre-training models like RoBERTa.
Key Insights and Ideas
- XLM-R Dominance: XLM-RoBERTa, a multilingual transformer model, consistently outperformed other models, especially in its supervised form, highlighting its strength in handling diverse linguistic inputs.
- Semi-Supervised Learning's Potential: SSL showed promising results, particularly for sentiment analysis, demonstrating its ability to boost performance and reduce the reliance on expensive human annotations, a critical factor for low-resource languages.
- AfriBERTa's Underperformance: Despite being specifically designed for African languages, AfriBERTa did not perform as well as general multilingual models like XLM-R, suggesting that current architectures might not fully capture the nuances of African linguistic contexts or that more specialized training data is needed.
- Emotion Detection Challenges: Emotion detection proved more challenging than sentiment analysis across all models, indicating the complexity of identifying subtle emotional cues in code-switched text.
- Bias Towards Neutral Sentiment: A common observation was that models tended to predict neutral sentiment, suggesting a potential bias in the data or the models' learning process.
Key Results
- Sentiment Analysis: XLM-R supervised achieved the highest accuracy (69.2%) and F1 score (66.1%). XLM-R semi-supervised also performed strongly (67.2% accuracy, 64.1% F1). AfriBERTa models showed the lowest performance.
- Emotion Analysis: DistilBERT supervised led in accuracy (59.8%) and F1 score (31%). mBERT semi-supervised followed closely (59% accuracy, 26.5% F1). AfriBERTa models again performed the worst.
- Specific Emotion Performance: XLM-R excelled in anger and love, while all models struggled with fear. Empathy was a strong point for XLM-R supervised (F1 of 76.8%), but sadness and surprise were challenging.
Limitations and Future Work
- Subjectivity and Label Bias: The authors acknowledge the inherent subjectivity in sentiment and emotion analysis, which can lead to inconsistencies in labeled data. They plan to publicly share the dataset to address this.
- Cost of Annotation: Obtaining high-quality labeled data, especially from native speakers, is expensive and time-consuming, posing a challenge for low-resource NLP.
- LLM Ethical Considerations: The authors recognize the ethical implications of using large language models.
- Future Work: The authors aim to enhance model performance by expanding human-labeled datasets, exploring other semi-supervised approaches (like few-shot learning), and experimenting with different model architectures and hyperparameter tuning.
Summary updated