IEST: WASSA-2018 Implicit Emotions Shared Task

Roman Klinger, Orphée De Clercq, Saif Mohammad, Alexandra BalahurEMNLP 2018

TL;DR

This paper introduces the Implicit Emotion Shared Task (IEST), a novel benchmark for detecting emotions in text where the explicit emotion word is masked, forcing models to infer emotions from context. The task highlights the challenge of understanding subtle emotional cues and reveals that automated systems can outperform human annotators in this nuanced task.

What Problem Does This Paper Address?

Traditional emotion recognition in text often relies on the presence of explicit emotion words (e.g., "happy," "sad"). However, human communication is rich with implicit emotional expressions where emotions are conveyed through descriptions of situations, events, or reactions, without directly stating the emotion. Existing datasets and tasks often fail to capture this subtlety, leading to models that might struggle when explicit emotion cues are absent. The IEST task aims to bridge this gap by creating a dataset and a shared task specifically designed to test a system's ability to infer emotions from context alone, mimicking how humans often understand emotions in real-world scenarios.

Methods

The core of the IEST task is its unique approach to data creation and evaluation. The researchers leveraged Twitter data, specifically tweets containing an emotion word followed by "that," "when," or "because." This structure suggests that the following phrase describes the cause or context of the emotion. Crucially, the emotion word itself was then masked (replaced with [#TARGETWORD#]), forcing participating systems to predict the emotion based solely on the surrounding text. This method is a form of "distant supervision," using the implicit structure of the tweets to automatically label the data. The evaluation metric used was macro F1-score, which balances precision and recall across all emotion classes.

Datasets

The primary dataset used for the IEST task was compiled from Twitter. Tweets were collected based on specific patterns involving emotion words and conjunctions. After masking the emotion word, the dataset was split into training (80%), trial (5%), and test (15%) sets, with careful stratification to ensure a balanced distribution of the six target emotion classes: anger, disgust, fear, joy, sadness, and surprise. This dataset is a novel benchmark specifically designed for implicit emotion detection.

Key Insights and Ideas

  • The Power of Context: The task demonstrates that context is paramount for understanding emotions, especially when explicit emotion words are absent. Systems had to learn to associate specific situations or descriptions with particular emotional responses.
  • Automation Outperforms Humans (in this specific task): Surprisingly, the study found that automated systems achieved higher performance than human annotators on the IEST task. This suggests that machines, by analyzing subtle textual patterns and word co-occurrences, might be able to pick up on linguistic clues that humans overlook or don't consciously process.
  • Challenges in Implicit Emotion Recognition: Despite the success of automated systems, the task still presented significant challenges, particularly for emotions like disgust and surprise, which showed lower prediction accuracy.
  • Distant Supervision as a Scalable Solution: The use of distant supervision, by leveraging existing linguistic structures (emotion word + conjunction), proved to be an effective method for creating a large-scale dataset for this nuanced task.

Key Results

The IEST shared task saw submissions from 30 teams, with the top-performing systems achieving macro F1 scores up to 71%. The baseline system (Max-Ent bag of words and bigrams) achieved a macro F1 score of 59.88%. The best system, "Amobee," employed an ensemble of transformer decoder models and transfer learning. A human evaluation study on a subset of the data yielded a macro F1 score of 45%, significantly lower than the automated systems. The results also indicated that joy was the most accurately predicted emotion, while disgust and surprise were the most challenging.

Limitations and Future Work

The authors acknowledge that the distant supervision approach, while scalable, might introduce inconsistencies into the dataset. Future work includes using crowdsourcing to improve data preprocessing and address potential issues with the dataset. A key future research direction is to encourage "introspection" of the models to understand how they infer emotions from subtle descriptions, rather than just associating words. The paper also suggests exploring transfer learning from existing corpora to the IEST data and vice versa, and applying the learned knowledge to other NLP tasks like argument mining and sentiment analysis.

Summary updated

BibTeX
@inproceedings{klinger2018iest,
  author = {Roman Klinger and Orphée De Clercq and Saif Mohammad and Alexandra Balahur},
  title = {{IEST: WASSA-2018 Implicit Emotions Shared Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure