Howard University Computer Science at SemEval-2023 Task 12: A 2-Step System Design for Multilingual Sentiment Classification with Language Identification

Saurav Aryal, Howard PrioleauACL 2023

TL;DR

This paper tackles the challenge of sentiment analysis for African languages, which are often underrepresented in NLP. The authors propose a novel two-step system that first identifies the language of a text and then uses a language-specific model for sentiment classification, achieving competitive or better performance than existing multilingual models. This approach highlights the effectiveness of specialized models for low-resource languages.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, is a crucial area in Natural Language Processing (NLP). However, most advancements and available datasets are heavily skewed towards high-resource languages like English. African languages, despite their vast diversity and importance, are significantly underrepresented in NLP research. This lack of resources makes it difficult to build effective sentiment analysis systems for these languages. The AfriSenti-SemEval Shared Task 12 aims to address this by providing new datasets for 12 African languages, but developing robust models for such a diverse set of languages, many of which have limited data, remains a significant challenge. The paper specifically addresses the need for models that can handle the nuances of individual African languages, rather than relying solely on general-purpose multilingual models that may not perform optimally.

Methods

The authors propose a two-step system designed for multilingual sentiment classification.

  1. Language Identification: The first step involves identifying the language of the input text. They chose AfroXLMR for this task due to its ability to adapt to unseen languages.
  2. Language-Specific Model Fusion: Once the language is identified, the system routes the text to a pre-trained, language-specific sentiment classification model. This approach is based on the finding that language-specific models often outperform general multilingual models, especially for low-resource languages.

For feature extraction, they employed two main methods:

  • Transformer-based models: These used tokenization with a maximum token length of 128.
  • Delta TF-IDF: This traditional feature-based approach was used for sentiment classification, with models like LightGBM, kNN, Random Forest, and SVM being trained and tuned.

The system design emphasizes modularity and extensibility, allowing for the easy integration of new languages and models.

Datasets

The paper utilizes the AfriSenti-SemEval Shared Task 12 datasets, which comprise 14 new datasets annotated for sentiment analysis across 12 different African languages: Hausa (HA), Yoruba (YO), Igbo (IG), Nigerian Pigdin (PCM), Amharic (AM), Algerian Arabic (DZ), Moroccan Arabic/Darija (MA), Swahili (SW), Kinyarwanda (KR), Twi (TWI), Mozambican Portuguese (PT), and Xitsonga (TS). These datasets are described as containing tweets, some of which are code-mixed or transliterated. The paper also references existing multilingual models trained on various datasets, including XLM-R (trained on over 2.5 terabytes of data).

Key Insights and Ideas

  • Language-specific models outperform multilingual models for low-resource languages: This is a central finding, suggesting that specialized models capture linguistic nuances better than general-purpose ones.
  • Delta TF-IDF is effective for low-resource languages: This traditional method requires less data and can be competitive with or even surpass larger transformer models, making it a valuable tool when data is scarce.
  • A two-step approach (Language ID + Language-Specific Model) is a robust strategy: This modular design allows for leveraging the strengths of both language identification and specialized sentiment classifiers.
  • Neutral sentiment is challenging: The paper notes that classifying neutral sentiment is often more difficult than positive or negative sentiment, and their approach performed well across all three classes.

Key Results

  • The proposed Language-Specific Model Fusion system achieved comparable or better classification performance than state-of-the-art models like AfriBERTa, AfroXLMR, and AfroLM on the AfriSenti-SemEval Task 12.
  • The Delta TF-IDF models, despite using less data, performed competitively and sometimes outperformed Transformer models on low-resource languages.
  • The Language Identification task achieved near-perfect classification results, demonstrating the effectiveness of the chosen model for this initial step.
  • For multilingual performance, their proposed approach outperformed current state-of-the-art Transformer models by at least 4% across all classification metrics.

Limitations and Future Work

  • Data scarcity: The authors acknowledge that more data is needed to enhance performance, and the use of pre-existing models might carry performance issues and biases.
  • Partnerships and native speakers: They suggest establishing partnerships with native speakers and increasing data access from African academic institutions to overcome limitations.
  • Scalability of language-specific models: As the number of languages grows, managing individual language-specific models becomes complex. The paper suggests that multilingual models might be a future solution for scalability.
  • Expertise limitation: The authors are not experts or speakers of the languages studied, which limits the depth of qualitative analysis.
  • Future work: The paper suggests that further improvements to individual language-specific models could enhance overall performance and allow for language-specific advancements. They also point towards multilingual models as a potential future direction for handling a large number of languages efficiently.

Summary updated

BibTeX
@inproceedings{aryal2023howard,
  author = {Saurav Aryal and Howard Prioleau},
  title = {{Howard University Computer Science at SemEval-2023 Task 12: A 2-Step System Design for Multilingual Sentiment Classification with Language Identification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure