Enhancing Cross-Lingual Emotion Detection with Data Augmentation and Token-Label Mapping

Jinghui Zhang, Yuan Zhao, Siqin Zhang, Ruijing Zhao, Siyu BaoACL 2024

TL;DR

This paper tackles the challenge of cross-lingual emotion detection, especially in low-resource languages, by employing data augmentation and a novel token-label mapping technique. Their approach significantly improved performance on the EXALT shared task, demonstrating effective emotion transferability across languages and better identification of emotion triggers.

What Problem Does This Paper Address?

Detecting emotions in text is crucial for understanding human sentiment, but doing so across different languages (cross-lingual emotion detection) is a significant hurdle. This is due to several factors:

  • Data Scarcity: Labeled data for emotion detection is often limited, especially for languages other than English.
  • Imbalanced Labels: Some emotions are expressed far more frequently than others, leading to biased models.
  • Cultural and Linguistic Differences: How emotions are expressed varies greatly across cultures and languages, making direct translation insufficient.
  • Figurative Language: Sarcasm, metaphors, and other non-literal language complicate emotion interpretation.
  • Model Opacity: Large pre-trained language models (PLMs) can be "black boxes," making it hard to understand why they make certain predictions, which is especially important for explainable AI.

The EXALT shared task at WASSA 2024 was designed to address these issues by focusing on emotion transferability and the identification of "emotion triggers" (words or phrases that signal an emotion).

Methods

The authors propose a two-pronged approach:

  1. Data Augmentation: To combat data scarcity and label imbalance, they use two techniques:
  2. Back-translation: Translating English training data into other languages (Dutch, French, Spanish, Russian) and then back into English. This introduces variations and potentially helps the model learn language-invariant features.
  3. Synonym Replacement: Replacing words with their synonyms using the NLTK WordNet corpus. This increases lexical diversity.
  4. Token-Label Mapping: For the sub-task of identifying emotion triggers (which words signal emotion), they introduce a novel method:
  5. Subword-Level Analysis: PLMs often process text at the subword level (e.g., "unhappiness" might be broken into "un", "happi", "ness"). The authors map the emotion label of a word to all its constituent subwords.
  6. Prediction Aggregation: During prediction, they take the maximum probability across all subwords of a word to determine the word's emotion label. This allows for finer-grained capture of emotional information. For binary trigger word detection, they use a threshold of 0.1 on these probabilities.

They utilize the XLM-ROBERTa-XL model, an encoder-only architecture, which they found suitable for capturing context in token-level tasks.

Datasets

The paper uses datasets provided for the EXALT shared task:

  • Task 1 (Emotion Detection): 5,000 training samples, 500 development samples, and 2,500 test samples. The training data is English.
  • Tasks 2 & 3 (Emotion Trigger Identification): 3,000 training samples, 300 development samples, and 832 test samples. The training data is English.
  • Test Set: The test data for all tasks includes five languages: Dutch, Russian, Spanish, English, and French.

Key Insights and Ideas

  • Data Augmentation is Key: Both synonym replacement and back-translation significantly improved performance, highlighting their effectiveness in enriching limited training data for cross-lingual tasks.
  • Subword-Level Nuance Matters: The token-label mapping approach, by considering subword information, allows for a more granular understanding of emotion triggers, leading to better performance in identifying them.
  • Encoder-Only PLMs Excel: For tasks requiring deep contextual understanding and token-level analysis, encoder-only architectures like XLM-ROBERTa-XL proved more effective than decoder-only models.
  • Cross-Lingual Transfer is Robust: Training on English data alone significantly boosted performance on other languages, suggesting that PLMs inherently possess some cross-lingual capabilities that can be fine-tuned for specific tasks like emotion detection.

Key Results

The authors achieved competitive results in the EXALT shared task:

  • Task 1 (Emotion Detection): 13th place.
  • Task 2 (Binary Trigger Word Detection): 1st place.
  • Task 3 (Numerical Trigger Word Detection): 2nd place.

Specifically, their augmented data approach, using both synonym replacement and back-translation, led to an F1 score of 0.5432 for Task 1 on the test set, outperforming models trained without augmentation. For Task 3, synonym replacement alone improved the F1 score to 0.6972.

Limitations and Future Work

  • Hyperparameter Assumption: The authors assume that optimal hyperparameters for XLM-ROBERTa-XL and XLM-ROBERTa-large are similar, which might not always hold true.
  • Fixed Threshold: A default threshold of 0.1 was used for identifying trigger words in Task 2. Future work could explore the impact of different thresholds.
  • Limited Augmentation: Only two data augmentation techniques were used. Exploring other methods or combinations could yield further improvements.
  • Future Directions: The authors suggest investigating the impact of different classification thresholds, exploring other data augmentation methods, and delving into more advanced model enhancement techniques like task analogy and model fusion for cross-lingual emotion detection.

Summary updated

BibTeX
@inproceedings{zhang2024enhancing,
  author = {Jinghui Zhang and Yuan Zhao and Siqin Zhang and Ruijing Zhao and Siyu Bao},
  title = {{Enhancing Cross-Lingual Emotion Detection with Data Augmentation and Token-Label Mapping}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure