TL;DR
This paper presents systems for cross-lingual emotion detection in tweets, achieving 3rd place in the WASSA 2024 Shared Task 2. They found that high-quality monolingual data and Chain-of-Thought prompting with trigger words are crucial for effective cross-lingual emotion detection, outperforming multilingual approaches in many cases.
What Problem Does This Paper Address?
Detecting emotions in social media, particularly tweets, is challenging due to their short, informal, and noisy nature, often filled with slang, abbreviations, and cultural nuances. This task becomes even more complex when dealing with multiple languages, as traditional NLP models struggle to transfer knowledge across linguistic boundaries. The WASSA 2024 Shared Task 2 specifically focuses on cross-lingual emotion detection, emphasizing model explainability and interpretability, especially with the rise of large language models (LLMs). This paper tackles the question of how to effectively build cross-lingual emotion detection systems, exploring the trade-offs between monolingual and multilingual data, and the impact of different modeling approaches like BERT-based classifiers and LLM prompting.
Methods
The authors developed two main systems:
- BERT-based Classifier: This system involves several stages:
- Data Augmentation: To address label imbalance in the official English training data, they augmented it with data from SemEval 2018 Task 1 and Sentiment140. They explored various combinations of monolingual (English back-translated) and multilingual data for this augmentation.
- Continued Pre-training: They adapted a pre-trained BERT model (TwHIN-BERT) by continuing its pre-training on the augmented datasets. This step aims to make the model more domain-specific.
- Fine-tuning: The pre-trained model was then fine-tuned on a balanced subset of the augmented data for the downstream emotion classification task.
- In-Context Learner (ICL) System: This system leverages LLMs (specifically GPT-4o) and employs a Chain-of-Thought (CoT) prompting strategy.
- Chain-of-Thought (CoT) with Trigger Words: The core idea here is to guide the LLM by explicitly identifying "trigger words" within a tweet that indicate a specific emotion. This approach enhances explainability by showing the reasoning process. They developed both English and multilingual CoT prompts.
- Model Selection: They experimented with different GPT models and found GPT-4o to be the most effective.
Datasets
The paper utilized the following datasets:
- WASSA 2024 Shared Task 2 Official Training Data: The primary dataset for the task.
- English SemEval 2018 Task 1: Used for data augmentation, particularly for creating a balanced training set by oversampling underrepresented emotion labels.
- Sentiment140: A large tweet dataset (1.6 million instances) used for augmenting the training data for continued pre-training.
- Back Translated Pre-train Data: Created by translating Arabic and Spanish SemEval 2018 Task 1 data into English, then back-translating it into multiple languages (Spanish, Russian, French, Dutch).
- Forward Translated Train Data: Created by translating the Official Train data into four other languages.
- Subsampled Multilingual Data: A subset of the Forward Translated Train data, balanced to match the distribution of the English Balanced Train set.
Key Insights and Ideas
- Quality over Quantity for Monolingual Data: The study suggests that high-quality, in-domain monolingual data (like their English back-translated data) can be more effective for cross-lingual tasks than simply using larger amounts of multilingual data, especially when the multilingual data isn't carefully curated.
- Chain-of-Thought with Trigger Words for Explainability: Explicitly identifying trigger words within tweets for LLMs significantly improves performance and provides a clear, interpretable reasoning path for emotion detection.
- Multilingual Continued Pre-training Can Be Harmful: Counter-intuitively, the authors found that multilingual continued pre-training sometimes degraded performance compared to monolingual pre-training, suggesting that careful alignment of data distribution is more critical than language diversity in this phase.
- Domain Adaptation is Key: Continued pre-training on domain-specific data (like tweets) is crucial for adapting general-purpose language models to the nuances of tweet-based emotion detection.
- Translation of Test Data Matters: Translating dev and test sets into English consistently improved performance, even for models that were pre-trained on multilingual data, highlighting the dominance of English in many pre-training corpora.
Key Results
- The authors' best-performing model, a GPT-4o system using English Chain-of-Thought prompts with trigger words (EngCoT), achieved 3rd place overall in the WASSA 2024 Shared Task 2 leaderboard with an F1 score of 0.6015.
- The BERT-based classifier, after continued pre-training with 130k back-translated data, outperformed models trained with larger amounts of data (1M and 1.7M), emphasizing the importance of data similarity.
- In the BERT-based classifier, the English back-translated model ("Balanced Train") performed better than the multilingual model ("Subsampled") under the same training data size.
- For prompting-based methods, EngCoT consistently outperformed multilingual CoT prompts (MulCoT) in terms of F1 and Precision, though MulCoT achieved slightly better recall.
Limitations and Future Work
- Google Translation API Dependency: The quality of their augmented data is heavily reliant on the Google Translation API, which is not deterministic.
- Limited Hyper-parameter Search: They did not perform extensive hyper-parameter tuning for continued pre-training, which could potentially improve classifier performance.
- Closed-Source Models: The use of proprietary LLM APIs (like GPT-4o) limits reproducibility and access for researchers.
- Non-Deterministic LLM Outputs: LLM outputs can be non-deterministic and sensitive to prompt variations.
- Sentiment vs. Emotion Distinction: Treating Sentiment140 data as purely "within-task" for emotion detection might be an oversimplification, as sentiment and emotion are related but distinct tasks.
- Future Work: The authors suggest further research into understanding the underlying mechanisms of in-context learning and its impact on performance, as well as exploring methods to convert noisy tweet tokens (like emojis) into more meaningful representations. They also note that the optimal number of CoT steps might vary by model and dataset.
Summary updated