Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic

Faiza BelbachirACL 2023

TL;DR

This paper tackles the challenge of sentiment analysis for Algerian Arabic tweets, a low-resource language variant. The authors propose a system combining the MARBERT language model with custom-built lexicons for subjective terms and emoticons, achieving a respectable F1 score of 0.70 and ranking 9th out of 30 participants in the SemEval-2023 Task 12.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, is crucial for understanding public opinion. While significant progress has been made for well-resourced languages like English, analyzing low-resource languages, especially dialectal variants, remains a significant challenge. Algerian Arabic, spoken by millions, presents unique difficulties due to its linguistic variations, lack of standardized orthography, and the informal nature of social media communication (e.g., code-switching, emojis). Existing models trained on standard Arabic often struggle with these dialectal nuances, necessitating specialized approaches.

Methods

The core of the proposed system is the MARBERT language model, a transformer-based model pre-trained on a massive corpus of Arabic tweets, making it particularly adept at understanding Arabic dialectal language. To address the limitations of MARBERT on certain Algerian dialectal sentiment terms, the authors implemented a lexical filtering approach as a post-processing step. This involved creating two custom lexicons:

  1. Subjective Terms Lexicon: Manually curated from Algerian Arabic tweets, focusing on terms that carry strong opinion but might be underrepresented in the MARBERT model.
  2. Emoticon Lexicon: Leveraging the significant role of emojis in online communication, this lexicon captures the sentiment associated with various emoticons.

The system's pipeline involves pre-processing tweets (including normalization of Arabic letters), feeding them into MARBERT for initial sentiment prediction, and then applying the lexicons to refine predictions, particularly for tweets initially classified as "neutral."

Datasets

The study utilized the AfriSenti-SemEval shared task corpus for Task 12 (monolingual opinion detection in Algerian Arabic). This corpus consists of tweets collected from Twitter. The authors created their own internal training and development sets by splitting the provided data. Specifically:

  • Internal Training Set: 1858 annotated tweets.
  • Internal Development Set: 207 annotated tweets.
  • Labeled Test Collection: Not provided to participants, implying evaluation was done on a separate, unseen test set.

Key Insights and Ideas

  • MARBERT's Strength for Dialects: The MARBERT model, pre-trained on a large Arabic tweet corpus, demonstrates superior performance compared to standard Arabic models like Arabic-BERT and CAMELBERT for Algerian dialect sentiment analysis.
  • Lexicons as Crucial Augmentation: Manually curated lexicons of subjective terms and emoticons significantly improve sentiment analysis performance, especially for ambiguous or neutral-classified tweets. This highlights the value of domain-specific linguistic resources.
  • Handling Dialectal Nuances: The paper underscores the importance of addressing specific linguistic features of dialectal Arabic, such as letter variations and code-switching, which are often overlooked by models trained on classical Arabic.
  • Emoticons as Sentiment Indicators: The study validates the significant contribution of emoticons to conveying sentiment in social media, making their inclusion in sentiment analysis models a valuable strategy.

Key Results

The system achieved an F1 score of 0.70 on the test data, securing 9th place out of 30 participating systems in the SemEval-2023 Task 12. The authors also found that their model performed better at identifying negative and positive sentiments than neutral ones, likely due to the distribution of classes in the dataset. The introduction of lexical filtering improved the F1 score by approximately 0.4% compared to using MARBERT alone.

Limitations and Future Work

The authors acknowledge that their system's performance could be further enhanced by:

  • Enriching Lexicons: Expanding the subjective terms and emoticon lexicons with more entries.
  • Exploring Other Models: Investigating different transformer-based models or combining MARBERT with other approaches.
  • Cross-lingual Transfer: Utilizing subjective lexicons from other languages.

They also note that while MARBERT outperformed other language models, the F1 score for neutral documents was notably lower than for positive and negative ones, indicating room for improvement in classifying neutral sentiments. The paper also mentions that SVM with TF-IDF and LSTM models yielded worse results, reinforcing the effectiveness of transformer-based approaches for this task.

Summary updated

BibTeX
@inproceedings{belbachir2023foul,
  author = {Faiza Belbachir},
  title = {{Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure