TL;DR
This paper systematically compares various sentiment and emotion classification methods in low-resource settings, finding that traditional rule- and dictionary-based approaches often rival or outperform modern few-shot and prompting methods, especially for emotion classification. Surprisingly, these older methods also demonstrate greater robustness to domain shifts.
What Problem Does This Paper Address?
The field of sentiment and emotion analysis has seen a surge in sophisticated deep learning models, particularly large language models (LLMs). However, these advancements are often evaluated in "optimal" settings with abundant training data. This paper addresses the critical gap of understanding how these methods perform when data is scarce, a common scenario in real-world applications and for less common languages. The authors aim to provide a systematic comparison of diverse approaches, from classic rule-based systems to cutting-edge LLM prompting, to reveal their strengths and weaknesses in low-resource environments, including out-of-domain and cross-lingual scenarios.
Methods
The paper compares several categories of methods:
- Supervised: Standard fine-tuning of models like DistilBERT and RoBERTa on varying amounts of training data (200, 100, 20 examples per class) to establish an upper bound.
- Few-shot: Similar to supervised but with extremely limited training data, exploring how well models generalize from a handful of examples.
- Prompting: Utilizing LLMs (OPT and FlanT5 models) by providing them with a few examples within a natural language prompt to elicit predictions. This leverages the LLM's inherent knowledge without explicit fine-tuning.
- Rule-based: Employing systems like SoCal, which rely on pre-defined rules and dictionaries to classify sentiment, requiring significant initial effort but no retraining.
- Dictionary-based: Using pre-compiled sentiment lexicons (e.g., HuLiu, NRC) to score texts based on word polarity.
- Dictionary Induction: Automatically creating sentiment lexicons using methods like SemAxis, which leverage word embeddings and a small set of seed words. This is explored for both sentiment and emotion.
- Cross-lingual: Evaluating multilingual LLMs (XLM-ROBERTa) for zero-shot cross-lingual transfer, where models trained on a high-resource language are tested on low-resource languages.
Datasets
The study utilizes a variety of datasets:
Sentiment Datasets (8 languages): * MPQA (English) * SemEval 2013 (English) * OpeNER (English, Spanish) * GermEval 2017 (German) * ASTD (Arabic) * NoReC (Norwegian) * MultiBooked (Basque) * NArabizi (Arabic transliterated script) * Maltese (Maltese)
Emotion Datasets (English): * SSEC (Stanford Sentiment Treebank) * EnISEAR
Key Insights and Ideas
- The enduring power of traditional methods: In low-resource settings, especially for emotion classification, classic rule- and dictionary-based methods perform surprisingly well, often matching or exceeding few-shot and prompting LLM approaches.
- Robustness to domain shift: Rule- and dictionary-based methods are significantly more resilient to changes in domain compared to supervised and few-shot methods, which tend to degrade substantially when tested on data from a different domain than they were trained on.
- Prompting's promise, but with caveats: While prompting LLMs shows promise, its effectiveness can be inconsistent, and its availability and performance for languages other than English lag behind.
- Dictionary induction as a strong contender: Automatically generated dictionaries through methods like SemAxis offer competitive performance, sometimes even outperforming hand-crafted lexicons, especially when starting with a small seed set.
- Cross-lingual models are powerful, but not a silver bullet: XLM-ROBERTa demonstrates strong cross-lingual capabilities, but its performance can be lower than dictionary induction in some cases, and it can be sensitive to the source language corpus.
Key Results
- Sentiment Classification (Low-Resource): For binary sentiment classification with very few examples (e.g., 100 per class), fine-tuning a model offers competitive in-domain performance. However, for even fewer examples, rule-based methods (SoCal) outperform. Few-shot-200 achieved the best in-domain performance (80.9 Macro F1), but suffered the largest domain transfer loss.
- Sentiment Classification (Domain Transfer): Supervised models achieved the best in-domain (91.2) and out-of-domain (84.6) performance with the smallest transfer loss (13.3). Prompting, rule-based, and dictionary-based approaches showed more consistent performance across domains.
- Emotion Classification (Low-Resource): Prompting (FlanT5) achieved the best performance (57.7 Macro F1), followed closely by dictionary induction (54.4) and Few-shot-200 (54.1). Dictionary-based methods and simpler few-shot approaches performed poorly.
- Cross-lingual Sentiment Classification: XLM-ROBERTa models generally outperformed dictionary induction approaches by 10-20 pp, achieving an average of 71.5 Macro F1 across languages, compared to 66.3 for prompting and 52.7 for dictionary induction.
Limitations and Future Work
- Binary vs. Multi-label: The study primarily focused on binary sentiment classification for easier cross-lingual comparison, acknowledging this simplifies the task. Emotion classification was multi-label but handled by treating each label independently.
- Dataset Diversity: While multiple datasets were used, they come from various domains, and controlling for this effect across all languages was challenging.
- Method Selection: The paper selected a representative subset of methods for each approach, acknowledging that other relevant methods might exist.
- Single Run Results: For supervised models, results were reported for a single run, not an average over multiple runs, though this was compensated by averaging across datasets and methods.
Future Work Suggestions: * Expand comparisons to include other dictionary induction methods, such as cross-lingual propagation. * Investigate how to effectively expand LLM prompting to new languages and tasks. * Compare multi-lingual few-shot approaches, which can leverage a few examples in the target language. * Further research into the importance of model size for low-resource performance, as findings were mixed.
Summary updated