Findings of the WASSA 2024 EXALT shared task on Explainability for Cross-Lingual Emotion in Tweets

Aaron Maladry, Pranaydeep Singh, Els LefeverACL 2024

TL;DR

This paper introduces the WASSA 2024 EXALT shared task, focusing on cross-lingual emotion detection and the identification of "trigger words" that evoke those emotions in tweets. The task highlights the challenges and progress in making emotion analysis models understandable and transferable across languages, with top systems achieving macro F1-scores of 0.629 for emotion detection and 0.616 for trigger word detection.

What Problem Does This Paper Address?

Understanding and explaining emotions in text is a crucial but challenging task in Natural Language Processing (NLP). While emotion detection itself has been studied, this paper tackles a more complex problem: cross-lingual emotion detection with explainability. Specifically, it addresses the difficulty of accurately identifying emotions in tweets across different languages (Dutch, Russian, Spanish, English, and French) when training data is primarily available in English. Furthermore, it aims to make these emotion detection systems more transparent by identifying the specific words within a tweet that are responsible for conveying the detected emotion. This is particularly important in an era of increasingly complex "black-box" models, where understanding why a model makes a certain prediction is as vital as the prediction itself.

Methods

The WASSA 2024 EXALT shared task was structured into two main sub-tasks:

  • Emotion Detection (Task 1): This involved classifying tweets into one of six emotion labels: Love, Joy, Anger, Fear, Sadness, and Neutral. The emotion label set was derived from existing research and adapted for social media text.
  • Trigger Word Detection (Task 2): This task required identifying the specific words or phrases within a tweet that evoke the predicted emotion. This serves as a form of explainability, highlighting the textual evidence for the emotion classification.

Participating systems were encouraged to develop cross-lingual approaches, leveraging the English training data to perform well on the other target languages. For trigger word detection, systems could approach it as a binary token classification task or by generating numerical importance scores for each word.

Datasets

The paper introduces a newly constructed multilingual benchmark dataset for cross-lingual emotion in tweets.

  • Training Data: Primarily English tweets (5000 for emotion detection, 3000 for trigger word detection).
  • Development Data: Multilingual (500 for emotion detection, 300 for trigger word detection).
  • Test Data: Multilingual (2500 for emotion detection, 832 for trigger word detection).

The tweets were collected using the Twitter API and filtered using stop words to minimize topical or emotional bias. A "discard" label was introduced for tweets that were difficult to interpret or were likely spam, and these were excluded from the final datasets. The annotation process involved detailed guidelines and was evaluated for inter-annotator agreement, yielding moderate scores for emotion and fair agreement for trigger words, reflecting the subjective nature of the task.

Key Insights and Ideas

  • Cross-lingual transfer is challenging but feasible: Despite the reliance on English training data, participating systems demonstrated that emotion detection and trigger word identification can be transferred to other European languages.
  • Explainability is crucial for trust: Identifying "trigger words" is a significant step towards making emotion detection systems more interpretable, moving beyond simple classification to understanding the textual basis for predictions.
  • Subjectivity impacts annotation and evaluation: The inherent subjectivity of emotion and trigger word identification leads to moderate agreement scores, highlighting the need for robust evaluation metrics and careful dataset construction.
  • Language-specific performance varies: Surprisingly, English was not consistently the best-performing language; Spanish emerged as the top-performing language for many systems.
  • Fear is a difficult emotion to detect: The class imbalance for "Fear" in the training data contributed to it being the hardest emotion to label accurately.

Key Results

The shared task saw participation from 14 teams for emotion detection and 6 for trigger word detection.

  • Emotion Detection: The top-performing system achieved a macro F1-score of 0.6295. All 14 teams outperformed the baseline score of 0.4476.
  • Trigger Word Detection: The best system for binary trigger word detection achieved a macro F1-score of 0.6158. For numerical trigger word detection, the top system achieved an API-score of 0.7052.

These results indicate that while progress has been made, cross-lingual emotion detection and explainability remain challenging areas.

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-annotator labels: Due to resource constraints, only single annotators were used for most labels, limiting the perspectives captured.
  • No multi-label classification: Tweets with mixed emotions were either assigned a primary emotion or discarded, potentially losing valuable data.
  • European language focus: The study is limited to European languages, which are relatively closely related and share the Latin script.

Future work suggested includes:

  • Expanding to more diverse languages: Investigating cross-lingual transfer to languages with different scripts or less relatedness.
  • Developing inherently explainable models: Moving beyond post-hoc explainability methods to models that are explainable by design.
  • Analyzing discarded tweets: Further investigating tweets that were labeled as "discard" to understand context-dependent emotions or mixed emotions.

Summary updated

BibTeX
@inproceedings{maladry2024findings,
  author = {Aaron Maladry and Pranaydeep Singh and Els Lefever},
  title = {{Findings of the WASSA 2024 EXALT shared task on Explainability for Cross-Lingual Emotion in Tweets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure