USI-IR at IEST 2018: Sequence Modeling and Pseudo-Relevance Feedback for Implicit Emotion Detection

Esteban Ríssola, Anastasia Giachanou, Fabio CrestaniEMNLP 2018

TL;DR

This paper introduces a novel approach for detecting emotions in tweets where the emotion word itself is removed, relying solely on context. They leverage a bidirectional LSTM (biLSTM) model enhanced with a pseudo-relevance feedback mechanism to iteratively refine its understanding of implicit emotions. While the proposed method shows promise, it also highlights the inherent challenges of this task and suggests avenues for future improvement.

What Problem Does This Paper Address?

The paper tackles the challenging problem of implicit emotion detection in social media, specifically on platforms like Twitter. Unlike traditional sentiment analysis that looks for explicit emotion words (e.g., "happy," "sad"), this task requires identifying emotions based solely on the surrounding context of a tweet, even when the emotion-defining word has been removed. This is crucial because people often express emotions indirectly, and understanding these nuances is vital for applications ranging from customer service to mental health monitoring. The Implicit Emotion Shared Task (IEST) at WASSA 2018 specifically focuses on this by removing emotion words from tweets and asking participants to predict the original emotion.

Methods

The core of the USI-IR system is a bidirectional Long Short-Term Memory (biLSTM) network. This choice is motivated by the sequential nature of text data, allowing the model to capture temporal dependencies and learn features directly from word sequences. The biLSTM is fed with word embeddings derived from a massive collection of tweets, providing a rich semantic representation of words.

The novel aspect of their approach is the pseudo-relevance feedback (PRF) schema. Inspired by Information Retrieval techniques, PRF assumes that the system's initial top predictions are correct. These high-confidence predictions are then treated as new, labeled examples to re-train the biLSTM network. This iterative process aims to refine the model's understanding of implicit emotions by exposing it to more contextually relevant examples, effectively "teaching" it to better infer emotions from subtle cues.

Datasets

The authors exclusively used the dataset provided by the IEST: WASSA-2018 Implicit Emotion Shared Task. This dataset consists of:

  • Training set: Approximately 153K tweets.
  • Development set: Approximately 10K tweets.
  • Test set: Approximately 28K tweets.

Each data instance includes a tweet and the emotion class of the word that was removed. No external datasets were used for training or development.

Key Insights and Ideas

  • Context is King (but challenging): The paper underscores that inferring emotions without explicit emotion words is a difficult but achievable goal, heavily reliant on nuanced contextual understanding.
  • Pseudo-Relevance Feedback for Emotion: Applying PRF, a technique traditionally used in information retrieval, to emotion detection is an innovative way to iteratively improve model performance by leveraging its own high-confidence predictions.
  • LIWC's Potential and Pitfalls: The exploration of the Linguistic Inquiry and Word Count (LIWC) tool revealed interesting patterns in word usage associated with different emotions (e.g., first-person pronouns in fear, causal words in anger). However, directly incorporating these findings into the model did not yield significant improvements, suggesting that more sophisticated integration methods are needed.
  • Hyperparameter Sensitivity: The performance of the PRF approach appears to be sensitive to the size of the expanded dataset and potentially requires re-optimization of hyperparameters like the number of epochs for each retraining step.

Key Results

The paper reports that the effectiveness of the proposed approach on the test set was not as satisfactory as expected. Specifically, they observed that performance diminishes as the size of the relevance set used for re-training increases. For instance, with a k% of 10, the accuracy was 63.72% and MacAvg was 0.60, while at k% of 50, accuracy dropped to 64.13% and MacAvg to 0.58. The authors attribute this to potential noise introduced by misclassified examples in the expanded set.

Limitations and Future Work

The authors acknowledge that the performance did not meet their expectations and suggest several avenues for future work:

  • Thresholding for PRF: Instead of simply taking the top-k predictions, defining a probability threshold to select more reliable examples for the expansion set could mitigate noise.
  • Hyperparameter Re-optimization: Given the change in training data size and content with PRF, re-optimizing hyperparameters like the number of epochs is crucial.
  • Enhanced LIWC Integration: Further exploration into more sophisticated ways to incorporate LIWC features into the network is recommended, as promising patterns were identified.
  • Exploring Other Features: The paper hints at the possibility of extracting other promising features that could help emphasize emotional differences in tweets.

Summary updated

BibTeX
@inproceedings{rissola2018usi,
  author = {Esteban Ríssola and Anastasia Giachanou and Fabio Crestani},
  title = {{USI-IR at IEST 2018: Sequence Modeling and Pseudo-Relevance Feedback for Implicit Emotion Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure