EmotiKLUE at IEST 2018: Topic-Informed Classification of Implicit Emotions

Thomas Proisl, Philipp Heinrich, Besim Kabashi, Stefan EvertEMNLP 2018

TL;DR

This paper introduces EmotiKLUE, a deep learning system for detecting implicit emotions in text, which significantly outperforms baseline methods by incorporating topic information. The system leverages independent representations of left and right contexts around emotion words, combined with LDA-derived topic distributions, to achieve state-of-the-art results on the Implicit Emotion Shared Task (IEST) 2018.

What Problem Does This Paper Address?

Detecting emotions in text is a well-established area, but inferring implicit emotions – those not explicitly stated but implied by the context – presents a significant challenge. The Implicit Emotion Shared Task (IEST) specifically targets this by asking participants to predict the emotion triggered by an emotion word, based on its surrounding text. The task organizers compiled a dataset of tweets where emotion words are followed by indicators of their cause (e.g., "because," "when"), and these emotion words are masked for prediction. The core problem is to build a model that can accurately infer the emotion from this contextual information, especially when the emotion word itself doesn't directly convey the feeling.

Methods

EmotiKLUE is a deep learning system built upon several key components:

  • Contextual Representations: The system learns independent representations for the text preceding (left context) and following (right context) the masked emotion word. This is achieved using unidirectional LSTM (Long Short-Term Memory) layers, which process the text from left-to-right and right-to-left, respectively.
  • Topic Modeling: To capture the thematic context of the tweets, the authors employ Latent Dirichlet Allocation (LDA) to derive a topic distribution for each tweet. They trained a Twitter-specific LDA model with 100 topics.
  • Integration of Topic Information: The paper explores three ways to integrate LDA topic distributions with the LSTM-based context representations:
  • As Additional Features: The LDA topic distribution is concatenated with the LSTM output before being fed into a dense layer.
  • As a Filter: The LSTM output is dimensionally reduced, and the LDA topic distribution is passed through a softmax layer. The outputs of these two layers are then combined using element-wise multiplication.
  • No LDA Topics: A baseline model without topic information.
  • Word Embeddings: The system utilizes pre-trained word embeddings. They experimented with both skip-gram and CBOW (Continuous Bag-of-Words) approaches, using 100- and 300-dimensional vectors.
  • Training Strategy: The model is trained using an Adam optimizer. They also explored a strategy of pre-training on a larger, in-house dataset and then fine-tuning on the official IEST training data.

Datasets

  • IEST 2018 Shared Task Data: This is the primary dataset, comprising over 190,000 tweets split into training (80%), trial (5%), and test (15%) sets. Emotion words are masked, and the context is provided.
  • ENCOW14: A resource used for additional pretraining.
  • In-house Collection of 114 Million Deduplicated English Tweets: This large dataset was used to train Twitter-specific word embeddings and LDA topic models.

Key Insights and Ideas

  • Topic Matters: The authors' core hypothesis that topic distribution influences emotion is supported, though the way it's integrated is crucial. Simply concatenating LDA features provided a slight benefit, while using LDA as a filter often hurt performance.
  • Contextual Embeddings are Key: The use of independent left and right context representations, similar to aspect-based opinion mining, proves effective.
  • Skip-gram Outperforms CBOW: Skip-gram embeddings consistently yielded better results than CBOW embeddings, suggesting their ability to capture nuanced semantic relationships.
  • Larger Embeddings Help: 300-dimensional embeddings generally outperformed 100-dimensional ones, especially for skip-gram.
  • Pretraining is Beneficial: Using a larger, in-house dataset for pretraining significantly improved performance, even if that dataset alone yielded poorer results. This highlights the value of diverse data for initial learning.
  • Implicit Emotion is Hard: The error analysis reveals significant confusions between semantically related emotions (e.g., anger/disgust, fear/surprise), indicating the inherent difficulty of inferring implicit emotions.

Key Results

  • Top Performance: EmotiKLUE achieved a macro average F₁ score of 67.13% on the test data, placing it tenth in the IEST 2018 shared task.
  • Improvement over Baseline: This score significantly outperformed a simple ML classifier baseline.
  • Post-Analysis Improvements: Further refinements, including using all 21 emotion words for pretraining and increasing hidden layer sizes to 300 units, led to an improved F₁ score of 68.10%.
  • Embedding Impact: Skip-gram embeddings outperformed CBOW by approximately 2.55% on average. 300-dimensional embeddings were better than 100-dimensional by about 1.19%.
  • Training Data Impact: The official training data yielded much better results than the in-house data alone (+12.97% on average). However, pretraining on the in-house data and then retraining on the official data provided a benefit of +1.96%.
  • LDA Impact: Using LDA topics as additional features offered a small positive effect (+0.08%) for skip-gram models but a negative effect (-0.24%) for CBOW models. Using LDA as a filter had a negative effect (-0.76%) overall.

Limitations and Future Work

  • Human Performance: The authors note that human performance on this task is estimated at around 45%, highlighting the inherent difficulty. The best submission achieved 71.45%, indicating there's still room for improvement.
  • Hyperparameter Tuning: The authors mention they did not fine-tune all hyperparameters of their model, suggesting this as a direction for future work.
  • LDA Contribution: The LDA topic model's contribution was found to be minimal, especially when used as a filter. Future work could explore more sophisticated ways to integrate topic information or different topic modeling techniques.
  • Ambiguity: The inherent ambiguity of tweets and the difficulty in disambiguating semantically similar emotions remain significant challenges.
  • Data Retrieval Issues: The error analysis mentions that not all tweets could be retrieved from the Twitter API, which can affect the accuracy of analyses based on triggerwords.

Summary updated

BibTeX
@inproceedings{proisl2018emotiklue,
  author = {Thomas Proisl and Philipp Heinrich and Besim Kabashi and Stefan Evert},
  title = {{EmotiKLUE at IEST 2018: Topic-Informed Classification of Implicit Emotions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure