CLP at SemEval-2019 Task 3: Multi-Encoder in Hierarchical Attention Networks for Contextual Emotion Detection

Changjie Li, Yun Xing2019 · DOI

TL;DR

This paper presents a hierarchical attention network (HAN) model for detecting emotions in conversational text, outperforming the baseline by a significant margin. The system leverages multi-encoder strategies, including ELMo embeddings and hand-crafted features, to capture nuanced emotional cues within conversational context.

What Problem Does This Paper Address?

Detecting emotions in text is challenging, especially in conversational settings where context is crucial. Unlike single-utterance emotion detection, conversations involve a dynamic interplay of messages, making it difficult to pinpoint the emotion of a specific turn. Traditional methods often struggle with low recall due to the absence of explicit emotion words. This paper tackles the problem of contextual emotion detection, specifically aiming to classify the emotion of a user's utterance within the context of the preceding two turns in a conversation.

Methods

The core of the proposed system is a Hierarchical Attention Network (HAN), chosen for its ability to model the hierarchical structure of conversations (words forming utterances, utterances forming conversations). The HAN architecture consists of two main layers:

  • Utterance Encoder: This layer processes individual user utterances. It combines three types of representations:
  • ELMo Embeddings: Deep contextualized word representations from a pre-trained language model.
  • Hand-craft Features: Features derived from emojis and emoticons, which are considered vital indicators of emotion.
  • Bi-LSTM-Attention: A Bidirectional Long Short-Term Memory network followed by an attention mechanism to capture contextual information and word importance within an utterance.
  • Conversation Encoder: This layer takes the representations of individual utterances and encodes the conversational context. It uses another Bi-LSTM with an attention mechanism to summarize the importance of each turn in the conversation.

The system also incorporates a robust preprocessing pipeline (using Ekphrasis) that includes tokenization, normalization of URLs/emails, annotation of uppercase/repeated/elongated words, manual spelling correction, and normalization of emojis/emoticons. Data augmentation was also employed by strategically relocating emojis and emoticons to create more training data.

Datasets

The primary dataset used is from SemEval-2019 Task 3, which provided:

  • 30,160 conversations for training.
  • 2,755 conversations for development.
  • 5,509 conversations for testing.

The authors also performed data cleaning and augmentation on the training set to improve model performance.

Key Insights and Ideas

  • Hierarchical Structure is Key: The HAN architecture effectively captures the nested nature of conversations, proving more effective than simpler models.
  • Multi-Modal Features Matter: Combining ELMo embeddings (semantic understanding) with hand-crafted emoji/emoticon features (explicit emotional cues) significantly boosts performance.
  • Context is King (Especially the Third Turn): The attention mechanism in the conversation encoder highlights the importance of specific turns. Notably, the third turn received the highest attention weight, aligning with the task's objective of predicting the third turn's emotion. The first turn also showed higher attention than the second, potentially due to both coming from the same user.
  • Preprocessing Pays Off: The Ekphrasis-based preprocessing pipeline, especially spelling correction and normalization of informal language, reduced the percentage of unknown words and improved overall performance.
  • Data Augmentation for Emotion: Augmenting data by repositioning emojis and emoticons proved beneficial, particularly for improving the detection of "Angry," "Sad," and "Happy" emotions.

Key Results

The proposed system achieved a micro-averaged F1 score of 0.7524 on the test set, outperforming the organizers' baseline model by 0.1656. The paper details the performance gains from incrementally adding components:

  • HAN alone improved the F1 score to 0.6935.
  • Adding ELMo and hand-craft features further increased it to 0.7102.
  • Preprocessing boosted it to 0.7459.
  • Data augmentation (DA) resulted in the best performance of 0.7524.

The system ranked 17th out of 165 teams on the Task 3 leaderboard.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations beyond the inherent challenges of contextual emotion detection. The data augmentation strategy, while effective, is specific to the assumption that emoji/emoticon positions don't alter sentence meaning.
  • Future Work: The authors suggest that their system's architecture and components could be applied to other conversational NLP tasks. Further exploration of more sophisticated data augmentation techniques for NLP could also be a promising direction.

Summary updated

Abstract

In this paper, we describe the participation of team ”CLP” in SemEval-2019 Task 3 “Con- textual Emotion Detection in Text” that aims to classify emotion of user utterance in tex- tual conversation. The submitted system is a deep learning architecture based on Hier- archical Attention Networks (HAN) and Em- bedding from Language Model (ELMo). The core of the architecture contains two represen- tation layers. The first one combines the out- puts of ELMo, hand-craft features and Bidi- rectional Long Short-Term Memory with At- tention (Bi-LSTM-Attention) to represent user utterance. The second layer use a Bi-LSTM- Attention encoder to represent the conversa- tion. Our system achieved F1 score of 0.7524 which outperformed the baseline model of the organizers by 0.1656.
BibTeX
@misc{li2019clp,
  author = {Changjie Li and Yun Xing},
  title = {{CLP at SemEval-2019 Task 3: Multi-Encoder in Hierarchical Attention Networks for Contextual Emotion Detection}},
  year = {2019},
  doi = {10.18653/v1/s19-2025},
}
Figure