UIO at SemEval-2023 Task 12: Multilingual fine-tuning for sentiment classification in low-resource Languages

Egil RønningstadACL 2023

TL;DR

This paper explores fine-tuning multilingual language models for sentiment analysis on low-resource African languages. The authors found that for languages with thousands of training samples, monolingual fine-tuning generally outperformed multilingual approaches, suggesting that adding more data can introduce noise for well-represented low-resource languages.

What Problem Does This Paper Address?

Sentiment analysis, a crucial NLP task for understanding opinions and moods, has historically been well-researched for high-resource languages like English. However, there's a significant gap in applying these techniques to low-resource languages, particularly those spoken in Africa. The SemEval-2023 Task 12 aimed to address this by providing datasets for sentiment analysis in 12 African languages, many of which are underrepresented in existing pre-trained language models. The challenge lies in effectively adapting powerful multilingual models to these languages, which may have varying degrees of similarity to languages seen during pre-training and can exhibit code-switching.

Methods

The core of this work involves fine-tuning pre-trained multilingual language models. The authors selected the XLM-Twitter (XLM-T) model, specifically the XLMT-sentiment variant, as their starting point. This model was already pre-trained on a large Twitter dataset and then fine-tuned on a multilingual Twitter sentiment classification dataset. They experimented with different fine-tuning strategies:

  • Monolingual Fine-tuning: Training the model separately on the data for each individual language.
  • Multilingual Fine-tuning: Training the model on a combined dataset of all 12 languages.
  • Language Family Fine-tuning: Grouping languages by their families (e.g., Afro-Asiatic, Volta-Congo, Creole) and training on these subsets.

They also included an mpnet-model for comparison, leveraging sentence-transformer concepts.

Datasets

The primary datasets used were provided by the SemEval-2023 Task 12. These included:

  • Subtask A: Training data for 12 individual African languages, with sample sizes ranging from approximately 800 to over 14,000 tweets per language.
  • Subtask B: A joint, multilingual training set containing data from all 12 languages.

The paper also references existing sentiment analysis datasets for Arabic, English, and Portuguese used in the pre-training of their chosen model.

Key Insights and Ideas

  • The "Curse of Multilinguality" in Low-Resource Settings: While multilingual models are powerful, adding more languages can sometimes degrade performance, especially when the target languages have a decent amount of training data. The paper suggests that for languages with thousands of samples, the in-language signal is strong enough, and additional multilingual data might introduce noise.
  • Monolingual Fine-tuning Dominance: For languages with sufficient training data (thousands of samples), fine-tuning the model on that language's specific data yielded the best results. This challenges the assumption that a purely multilingual approach is always superior for low-resource languages.
  • Importance of Pre-training Data: The performance of models was influenced by how well the target languages were represented in the original pre-training data of the LLM. Languages seen during pre-training or initial fine-tuning tended to perform better.
  • Surprising Underperformance of Nigerian Pidgin: Despite its supposed relatedness to English, Nigerian Pidgin performed poorly, a finding the authors could not definitively explain.

Key Results

  • Monolingual Fine-tuning Wins: In their subsequent analysis (Table 3), monolingual fine-tuning achieved the highest F1-scores for nearly all languages. For example, Moroccan Arabic/Darija reached 97.5% with in-language fine-tuning, significantly outperforming the multilingual approach (96.7%).
  • XLMT-sentiment as a Strong Baseline: The XLMT-sentiment model, fine-tuned on a multilingual Twitter sentiment dataset, served as a good starting point, achieving competitive results across many languages.
  • Swahili as an Exception: Swahili was the only language where multilingual fine-tuning outperformed monolingual fine-tuning in the initial experiments.
  • Overall Performance: The authors acknowledge that their submission's results were around average or lower, attributing this partly to their constraint on using only "base" size models and not leveraging external language resources.

Limitations and Future Work

  • Model Size Constraint: The authors were limited to using "base" size LLMs due to resource constraints, which might have impacted performance compared to larger models.
  • Limited Exploration of Language Family Groupings: While they explored language family subsets, the analysis was not exhaustive.
  • Few-Shot Learning Potential: The paper suggests that for languages with very few training samples (e.g., ten or a hundred), multilingual data might be more beneficial. Future work could explore this "few-shot" scenario to understand when multilingual data truly adds value.
  • Investigating Underperforming Languages: Further research is needed to understand why languages like Nigerian Pidgin performed poorly, despite expectations.
  • Ethical Considerations: The authors acknowledge the ethical implications of working with low-resource languages, emphasizing their intent to contribute to digital inclusion and language diversity, while also recognizing that their work may not be of direct, immediate benefit to the language users.

Summary updated

BibTeX
@inproceedings{rnningstad2023uio,
  author = {Egil Rønningstad},
  title = {{UIO at SemEval-2023 Task 12: Multilingual fine-tuning for sentiment classification in low-resource Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure