FIT BUT at SemEval-2023 Task 12: Sentiment Without Borders - Multilingual Domain Adaptation for Low-Resource Sentiment Classification

Maksim Aparovich, Santosh Kesiraju, Aneta Dufkova, Pavel SmrzACL 2023

TL;DR

This paper tackles the challenge of sentiment analysis for low-resource African languages by proposing a novel language-centric domain adaptation approach. Their method, using adversarial training with a modified Afro-XLM-Roberta model, significantly improves sentiment classification performance across multiple subtasks, demonstrating the effectiveness of treating languages and language families as domains for adaptation.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, is crucial for understanding public opinion and user feedback. However, applying this to low-resource languages, particularly those in Africa, is a significant challenge due to the scarcity of annotated data and language-specific resources. SemEval-2023 Task 12 specifically aimed to advance research in this area by providing the Afrisenti dataset, which contains tweets in 14 African languages. The core problem is how to build robust sentiment analysis models that can generalize well to these languages, especially when they are underrepresented in training data or entirely unseen during training (zero-shot scenarios). This is exacerbated by "domain shift," where models trained on one data distribution struggle with another, a common issue when dealing with diverse languages and dialects.

Methods

The authors propose a language-centric domain adaptation approach using adversarial training. Their system comprises two main components:

  • Generator Model: A small version of Afro-XLM-Roberta, a Transformer-based language model pre-trained on African languages, is used as the generator. This model takes input text and produces latent representations.
  • Discriminator Model: A feed-forward network acts as the discriminator. Its role is to distinguish between different "domains."

The adversarial training works in two phases:

  1. The discriminator learns to predict the correct domain (language or language family) based on the generator's latent representations.
  2. The generator is trained to produce representations that not only help predict the sentiment class but also "fool" the discriminator, making it uncertain about the true domain. This encourages the generator to produce language-agnostic representations.

The authors experimented with different strategies for the discriminator's input (token-level vs. sequence-level representations) and domain labels (individual languages vs. language families). They also explored the impact of an adversarial loss scaling factor.

Datasets

The primary dataset used is the Afrisenti dataset (Muhammad et al., 2023a), which was created for SemEval-2023 Task 12. This dataset consists of tweets in 14 African languages, labeled with positive, neutral, or negative sentiment. The languages span four major language families: Afro-Asiatic, English Creole, Indo-European, and Niger-Congo. The authors used this dataset for training, validation, and testing across three subtasks: monolingual, multilingual, and zero-shot sentiment classification.

Key Insights and Ideas

  • Language Families as Domains: Treating languages and their families as distinct domains for adversarial adaptation proved effective, especially for improving cross-lingual generalization to unseen languages.
  • Sequence-Level Discriminator Advantage: Using sequence-level representations (an average of token representations) for the discriminator yielded better performance than token-level inputs, suggesting that capturing broader semantic context is more beneficial.
  • Language Proximity Matters: Training on languages that are closely related within a language family tree can enhance sentiment analysis quality for low-resource languages, even in zero-shot settings.
  • Practical Efficiency: The proposed approach maintains computational efficiency in the prediction step, making it practical for real-world applications.

Key Results

The proposed domain adaptation approach consistently outperformed the baseline across most of the 15 tracks in the SemEval-2023 Task 12.

  • Weighted F1 Improvement: The method achieved improvements in weighted F1 scores for 13 out of 15 tracks, with a maximum increase of 4.3 points for Moroccan Arabic compared to the baseline.
  • Subtask C (Zero-Shot): For Tigrinya (an unseen language during pre-training), domain adaptation with language family labels showed strong performance. However, for Oromo (a seen language), the baseline performed better, suggesting a nuanced interaction between pre-training data and adaptation strategies.
  • Discriminator Labels: Language family-based labels were found to be superior to language-level labels in the cross-lingual setup for unseen languages.

Limitations and Future Work

The authors acknowledge several limitations:

  • Ad-hoc Language Mapping: The mapping of languages to language families was done heuristically. Future work could explore different mapping strategies and their impact on performance.
  • Discriminator Architecture: The discriminator was limited to a feed-forward network. Investigating more complex architectures like RNNs or Transformer decoders for the discriminator could yield further improvements.
  • Model Size and Script: The study focused on a single, smaller version of Afro-XLM-Roberta and did not consider other model sizes or the influence of language scripts. Future research could explore these aspects.

The paper suggests that further research is needed to fully understand the nuances of domain adaptation when based on linguistic information.

Summary updated

BibTeX
@inproceedings{aparovich2023fit,
  author = {Maksim Aparovich and Santosh Kesiraju and Aneta Dufkova and Pavel Smrz},
  title = {{FIT BUT at SemEval-2023 Task 12: Sentiment Without Borders - Multilingual Domain Adaptation for Low-Resource Sentiment Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure