UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection

Jakub Šmíd, Pavel Přibáň, Pavel KrálACL 2024

TL;DR

This paper presents a system for cross-lingual emotion detection and trigger word identification in tweets. By fine-tuning quantized large language models (LLMs) with low-rank adapters (LoRA) and leveraging machine translation, the system achieved top rankings in the WASSA-2024 Shared Task 2, demonstrating the effectiveness of these techniques for multilingual NLP tasks.

What Problem Does This Paper Address?

The paper tackles the challenge of understanding emotions expressed in text across different languages, specifically within the context of social media. While emotion detection in text is a well-established NLP task, extending it to multiple languages (cross-lingual) and identifying the specific words that evoke these emotions adds significant complexity. The WASSA-2024 Shared Task 2 aims to address this by requiring systems to: 1) detect one of six emotion labels in tweets across English, Dutch, Russian, Spanish, and French, and 2) pinpoint the exact words triggering these emotions, outputting this information in both binary and numerical formats. A key challenge is that training data is often available only in English, necessitating effective methods for transferring this knowledge to other languages.

Methods

The authors employed a multi-pronged approach combining several advanced NLP techniques:

  • Quantized LLMs with LoRA: For the emotion detection subtask, they fine-tuned a quantized version of the Orca 2 LLM. To manage the computational cost of fine-tuning such large models, they utilized Low-Rank Adaptation (LoRA). LoRA injects small, trainable "adapter" matrices into the pre-trained model, significantly reducing the number of parameters that need to be updated while keeping the original model frozen.
  • Multilingual Transformer Models: For the trigger word detection subtask, they leveraged established multilingual Transformer models like XLM-R and mT5.
  • Machine Translation: To overcome the English-centric nature of the training data, they extensively used machine translation. The English training data was translated into the other four target languages to create additional training sets.
  • Alignment-Free Label Projection: For the trigger word detection subtask, directly translating labels can be problematic due to token misalignment. The authors adopted an alignment-free approach where trigger words were marked with special symbols before translation, and then extracted from the translated text.
  • Trigger Word Switching: To further augment the training data for trigger word detection, they created new bilingual sentences by switching the trigger words between the source and target languages.

Datasets

The primary dataset used was provided by Maladry et al. (2024) for the WASSA-2024 Shared Task 2, containing tweets in five languages (English, Dutch, Russian, Spanish, and French). The authors also generated additional datasets through machine translation of the original English training data into the other four languages.

Key Insights and Ideas

  • The Power of Translated Data: The study strongly suggests that augmenting English-only training data with machine-translated data significantly boosts performance, especially for models like Orca 2 that are primarily pre-trained on English.
  • LoRA for Efficient LLM Fine-tuning: The use of LoRA proved crucial for making the fine-tuning of large, quantized LLMs computationally feasible, enabling them to achieve strong results without requiring massive GPU resources.
  • Translation Challenges for Trigger Words: Identifying trigger words is more sensitive to translation errors than general emotion detection. The alignment-free projection method and trigger word switching were key to mitigating these issues.
  • Model Specialization: While Orca 2 excelled with translated data for emotion detection, the multilingual XLM-R and mT5 models showed strong performance in trigger word detection, highlighting the strengths of different model architectures for specific tasks.

Key Results

The system achieved excellent performance in the WASSA-2024 Shared Task 2:

  • Emotion Detection: Ranked 7th overall, with the Orca 2 model showing a significant improvement (over 3% on test data) when fine-tuned on translated data.
  • Binary Trigger Word Detection: Ranked 3rd, with XLM-R achieving a test score of 59.19 when trained on a combination of all datasets.
  • Numerical Trigger Word Detection: Ranked 1st, with the mT5 model achieving the top test score of 70.52 when trained on the full combination of datasets, outperforming the second-best settings by 8%.

Limitations and Future Work

  • Reliance on Machine Translation: The effectiveness of the system is inherently tied to the quality of the machine translation used. Translation errors can negatively impact performance, particularly for tasks like trigger word detection.
  • Independent Subtask Solving: The authors acknowledge that solving the emotion detection and trigger word detection subtasks independently might not be optimal. Future work could explore methods for solving both subtasks simultaneously to potentially achieve more robust and explanatory models.
  • Unintended Biases: The paper notes that the language models used may inadvertently introduce biases related to race or gender due to their pre-training on large corpora.

Summary updated

BibTeX
@inproceedings{smid2024uwb,
  author = {Jakub Šmíd and Pavel Přibáň and Pavel Král},
  title = {{UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure