GunadarmaXBRIN at SemEval-2023 Task 12: Utilization of SVM and AfriBERTa for Monolingual, Multilingual, and Zero-shot Sentiment Analysis in African Languages

Novitasari Arlim, Slamet Riyanto, Rodiah Rodiah, Al Hafiz Akbar Maulana SiagianACL 2023

TL;DR

This paper presents the GunadarmaXBRIN team's participation in SemEval-2023 Task 12, focusing on sentiment analysis for African languages. They explored SVM and the AfriBERTa model, achieving competitive results, particularly with SVM + One Vs Rest with SMOTE, highlighting the challenges and nuances of low-resource language sentiment analysis.

What Problem Does This Paper Address?

Sentiment analysis (SA), a crucial Natural Language Processing (NLP) task, has historically been dominated by research on high-resource languages like English and Chinese. This leaves a significant gap in understanding and developing SA capabilities for low-resource languages, particularly those spoken in Africa. SemEval-2023 Task 12 directly addresses this by focusing on sentiment analysis for a diverse set of African languages, presenting challenges related to data scarcity, linguistic diversity, and the need for robust models that can handle monolingual, multilingual, and zero-shot scenarios.

Methods

The GunadarmaXBRIN team employed a combination of established and adapted techniques:

  • Support Vector Machine (SVM) with One Vs Rest (OvR): Given the multi-class nature of the sentiment analysis task (positive, neutral, negative), SVM, a binary classifier, was adapted using the OvR strategy. This involves training a separate binary classifier for each class against all other classes.
  • SMOTE (Synthetic Minority Oversampling Technique): To address class imbalance in the training datasets, particularly for subtask A, SMOTE was utilized. This technique generates synthetic samples for minority classes, helping to create a more balanced dataset for training.
  • AfriBERTa-large: This is a pre-trained multilingual model specifically designed for African languages. The team fine-tuned this model to adapt it to the sentiment analysis task. They also trained a custom AfriBERTa-large tokenizer to support languages not covered by the original model, using corpora from various sources.

Datasets

The research utilized datasets provided for SemEval-2023 Task 12, which comprises 14 African languages: Hausa, Yoruba, Igbo, Nigerian Pidgin, Amharic, Algerian Arabic, Moroccan Arabic/Darija, Swahili, Kinyarwanda, Twi, Mozambican Portuguese, Xitsonga (Mozambique Dialect), Tigrinya, and Oromo. These datasets were used for training, development, and evaluation across three subtasks:

  • Subtask A (Track 12): Monolingual sentiment classification (specifically Xitsonga).
  • Subtask B (Track 16): Multilingual sentiment classification across 12 languages.
  • Subtask C (Track 18): Zero-shot sentiment classification (specifically on Oromo).

Key Insights and Ideas

  • The challenge of low-resource languages: The paper underscores the significant difficulties in applying NLP techniques to languages with limited data and resources. Performance often lags behind high-resource languages, necessitating specialized models and techniques.
  • SVM's resilience with SMOTE: The SVM + One Vs Rest with SMOTE model demonstrated strong performance, particularly in the evaluation phase for Tracks 16 and 18. This suggests that traditional machine learning methods, when augmented with effective data balancing techniques, can remain competitive even against advanced transformer models in certain scenarios.
  • Development vs. Evaluation phase discrepancies: A notable observation was that the best-performing model in the development phase did not always translate to the best performance in the evaluation phase. This highlights the importance of robust evaluation and the potential impact of subtle differences in dataset characteristics between development and test sets.
  • Linguistic similarities and sentiment confusion: The paper points out that even languages from different families can share similar words, which can lead to sentiment confusion. This was observed in Hausa and Yoruba, where similar-sounding words could carry different sentiments, impacting classification accuracy.
  • AfriBERTa-large's potential and limitations: While AfriBERTa-large is a promising model for African languages, its performance was sometimes surpassed by SVM, especially in the evaluation phase. This could be attributed to factors like additional corpora used for tokenizer training and hyperparameter choices.

Key Results

The team achieved the following highest weighted F1-Scores:

  • Track 12 (Xitsonga, Subtask A): 50.56% (SVM + One Vs Rest)
  • Track 16 (Multilingual, Subtask B): 65.14% (SVM + One Vs Rest)
  • Track 18 (Zero-shot Oromo, Subtask C): 33.49% (SVM + One Vs Rest with SMOTE)

While these results were competitive, they were generally lower than the top-ranking teams, indicating the task's complexity. The SVM + One Vs Rest (with or without SMOTE) models generally outperformed their AfriBERTa-large implementation in the evaluation phase.

Limitations and Future Work

  • Underperformance compared to baselines: The authors acknowledge that their models did not perform as well as or better than the baseline models provided by the task organizers.
  • Hyperparameter tuning: They suggest that further exploration of hyperparameter tuning for pre-trained models could improve performance.
  • Latin script tweets: The presence of tweets written in Latin script (often code-mixed) presents a challenge. Future work could involve exploring translation techniques (e.g., Arabic to Latin) to better handle such instances.
  • Deeper analysis of misclassifications: The paper suggests further investigation into misclassified tweets, particularly those containing mixed positive and negative sentiment cues or code-switching, to better understand classification errors.
  • Dataset characteristics: The authors hypothesize that dataset characteristics, such as the distribution of positive vs. negative instances between development and evaluation sets, might explain performance discrepancies. Further research is needed to validate this.

Summary updated

BibTeX
@inproceedings{arlim2023gunadarmaxbrin,
  author = {Novitasari Arlim and Slamet Riyanto and Rodiah Rodiah and Al Hafiz Akbar Maulana Siagian},
  title = {{GunadarmaXBRIN at SemEval-2023 Task 12: Utilization of SVM and AfriBERTa for Monolingual, Multilingual, and Zero-shot Sentiment Analysis in African Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure