Sentylic at IEST 2018: Gated Recurrent Neural Network and Capsule Network Based Approach for Implicit Emotion Detection

Prabod Rathnayaka, Supun Abeysinghe, Chamod Samarajeewa, Isura Manchanayake, Malaka WalpolaEMNLP 2018

TL;DR

This paper presents a novel approach for detecting implicit emotions in tweets, where explicit emotion words are removed. By combining Gated Recurrent Units (GRUs) with Capsule Networks, the authors achieve a strong performance on the WASSA 2018 Implicit Emotion Shared Task, demonstrating the effectiveness of this architecture for capturing nuanced emotional context.

What Problem Does This Paper Address?

The paper tackles the challenge of implicit emotion detection in social media. Unlike traditional sentiment analysis that relies on explicit emotion-laden words (e.g., "happy," "sad"), this task requires inferring emotions from the surrounding context when those explicit cues are deliberately removed. This is crucial because people often express emotions indirectly, and understanding these subtle cues is vital for deeper analysis of online discourse. Existing methods that rely on noisy features like emojis or hashtags can be unreliable, highlighting the need for more robust approaches that can learn from contextual linguistic patterns.

Methods

The core of the proposed system, "Sentylic," is a deep learning architecture that combines two key components:

  • Gated Recurrent Units (GRUs): These are used as the initial layer to process the input tweet. GRUs are a type of recurrent neural network (RNN) well-suited for sequential data like text, capable of learning long-term dependencies. The paper employs a bidirectional GRU (Bi-GRU), meaning it processes the text in both forward and backward directions to capture a more comprehensive understanding of the context.
  • Capsule Networks: Following the GRU layer, the output is fed into a Capsule Network. Capsule Networks, introduced by Sabour et al. (2017), are designed to preserve hierarchical relationships and spatial information, which the authors argue is lost in traditional pooling layers of Convolutional Neural Networks (CNNs). In this context, the Capsule Network aims to learn higher-level features and the emotional context from the representations generated by the GRU.

The model also incorporates:

  • Pre-trained Word Embeddings: Word2Vec embeddings are used to initialize the input layer, providing the model with rich semantic information about words.
  • Preprocessing Pipeline: A thorough preprocessing pipeline is implemented, including tweet tokenizing, word normalization (e.g., converting to lowercase, standardizing URLs and mentions), and spell correcting/word segmentation for hashtags.
  • Classification Layer: A final dense layer with a softmax activation function is used for classification into one of the six target emotions.

Datasets

The paper evaluates its approach on the dataset provided for the WASSA 2018 Implicit Emotion Shared Task. This dataset consists of tweets from which explicit emotion terms or their synonyms have been removed, requiring participants to predict the intended emotion. The authors do not mention creating or using any other datasets for training or evaluation.

Key Insights and Ideas

  • Capsule Networks for NLP: The paper explores the application of Capsule Networks, which have shown significant success in computer vision, to Natural Language Processing (NLP) tasks. The authors hypothesize that Capsule Networks can better preserve the hierarchical and relational features of text compared to traditional CNNs with pooling, which might discard crucial information.
  • Implicit Emotion Detection is Challenging: The results highlight the inherent difficulty of implicit emotion detection, particularly when emotions are nuanced or ambiguous, as seen in the example where anger and sadness are difficult to distinguish even for humans.
  • GRU-Capsule Synergy: The combination of GRUs for sequential context understanding and Capsule Networks for hierarchical feature extraction proves to be a powerful architecture for this specific task.

Key Results

The proposed "Sentylic" model achieved a macro-F1 score of 0.692, ranking 5th out of 30 contestants in the WASSA 2018 Implicit Emotion Shared Task. This represents a significant improvement of 0.155 over the baseline Maximum Entropy Model using bag-of-words and bigrams. The paper also presents a detailed breakdown of performance per emotion class, noting that "anger" had a lower F1-score, likely due to ambiguity in some examples.

Limitations and Future Work

  • Ambiguity in Emotion Expression: The authors acknowledge that certain expressions are inherently ambiguous, making it difficult to assign a single emotion even for human annotators. This ambiguity directly impacts the model's performance, particularly for emotions like anger.
  • Future Work: The authors suggest exploring the use of Capsule Networks for other NLP tasks, especially those where CNNs are currently employed. They also propose investigating the transfer learning of models trained on semi-automatically annotated datasets to human-annotated datasets.

Summary updated

BibTeX
@inproceedings{rathnayaka2018sentylic,
  author = {Prabod Rathnayaka and Supun Abeysinghe and Chamod Samarajeewa and Isura Manchanayake and Malaka Walpola},
  title = {{Sentylic at IEST 2018: Gated Recurrent Neural Network and Capsule Network Based Approach for Implicit Emotion Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure