Multilogue-Net: A Context-Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation

Aman Shenoy, Ashish SardanaACL 2020

TL;DR

This paper introduces Multilogue-Net, a novel RNN architecture for multi-modal emotion and sentiment analysis in conversations. It addresses the limitations of prior work by effectively capturing conversational context, inter-speaker dependencies, and the interplay between different modalities (text, audio, video), achieving state-of-the-art results on benchmark datasets.

What Problem Does This Paper Address?

Understanding human emotions and sentiments in conversations is crucial for many applications, from customer service bots to social media analysis. While multi-modal approaches (using text, audio, and video) offer a richer understanding than single modalities, existing methods struggle to:

  • Capture conversational context: They often treat each utterance in isolation, missing the flow and history of the dialogue.
  • Model inter-speaker dynamics: The emotional states of speakers and listeners, and how they influence each other, are often overlooked.
  • Fuse modalities effectively: Simply concatenating features from different modalities doesn't leverage their unique strengths or the relationships between them.

This paper aims to bridge this gap by proposing a model that explicitly models these contextual and inter-modal relationships.

Methods

Multilogue-Net employs a sophisticated Recurrent Neural Network (RNN) architecture designed to process multi-modal conversational data. Key components include:

  • Modality-Specific GRUs: For each modality (text, audio, video) and each participant, separate Gated Recurrent Units (GRUs) are used to encode the context (cGRU) and the speaker's state (sGRU).
  • The Context GRU (cGRU) captures the conversation's context by jointly encoding the current utterance's representation and the previous speaker's state.
  • The State GRU (sGRU) models the interlocutor's state, incorporating utterance representations and context-aware attention over past utterances.
  • Emotion GRU (eGRU): This GRU acts as a decoder, taking the state encoding from the sGRU and producing an emotion/sentiment representation for each modality. This allows for consistent representation across different tasks (emotion vs. sentiment).
  • Pairwise Attention Mechanism: This is a novel fusion technique. Instead of a single fusion step, it calculates attention over pairs of modality emotion representations (e.g., text-audio, text-video, audio-video). This allows the model to dynamically weigh the importance of different modality combinations for prediction.
  • Final Prediction Layer: This layer takes the fused representations and outputs either a sentiment score (regression) or probabilities for different emotion classes (classification).

The model is designed to be flexible, allowing for different numbers of participants and modalities.

Datasets

The proposed model is evaluated on two benchmark multi-modal datasets:

  • CMU-MOSI (Multi-modal Opinion-level Sentiment Intensity): Contains videos with sentiment labels for utterances.
  • CMU-MOSEI (Multi-modal Opinion Sentiment and Emotion Intensity): A larger dataset with both sentiment and emotion labels for utterances.

The paper also notes that for CMU-MOSEI, they project the continuous sentiment labels into a two-class setup (positive/negative) for A2 accuracy, and also report results for continuous range prediction and 6-class emotion prediction.

Key Insights and Ideas

  • Text is King (for multi-modal performance): The authors consistently observe that including text features significantly boosts performance across modalities. Video and audio features seem to act as enhancements to the text-based understanding.
  • Context is Crucial: Explicitly modeling conversational context and inter-speaker states through dedicated GRUs leads to significant improvements over models that treat utterances independently.
  • Pairwise Attention for Smarter Fusion: The pairwise attention mechanism proves more effective than simple concatenation or other fusion methods, allowing the model to learn nuanced relationships between modalities.
  • Task Consistency with eGRU: Using a shared emotion GRU (eGRU) for both sentiment and emotion tasks, while having task-specific prediction layers, helps maintain consistency and allows for potential multi-task learning. The Euclidean distance analysis in Table 5 highlights that emotion representations are more task-dependent than state or context representations.

Key Results

Multilogue-Net consistently outperforms previous state-of-the-art models on both CMU-MOSI and CMU-MOSEI datasets across various metrics (accuracy, F1 score, MAE, r).

  • CMU-MOSI: Achieved an F1 score of 80.10 for Text+Audio+Video, outperforming DialogueRNN (79.48) and MMMU-BA (82.31).
  • CMU-MOSEI: Demonstrated strong performance, particularly on emotion labels where it significantly outperformed existing models, likely due to fewer prior works reporting results on these specific labels. For sentiment, it achieved an A2 accuracy of 82.10 and F1 of 80.01 in the tri-modal setting.

The paper also shows that the fusion mechanism is beneficial, especially when text is involved, reinforcing the idea of text as a guiding modality.

Limitations and Future Work

  • Dataset Size: The authors theorize that the model's performance on CMU-MOSI might be limited by the relatively small number of training examples (93 conversations). Multilogue-Net's high capacity requires more data to learn robust representations.
  • Feature Extractors: While the model architecture is novel, the paper suggests that further improvements could be made by using more advanced feature extractors for each modality.
  • Extending Modalities and Participants: The model could be extended to include more modalities (e.g., physiological signals) and handle a larger number of participants. However, the lack of suitable datasets for such extensions is a current limitation.
  • Multi-task Learning: The paper hints at the potential of training multiple tasks in parallel using different eGRUs, which could further enhance performance and efficiency. This is left as future work.

Summary updated

BibTeX
@inproceedings{shenoy2020multilogue,
  author = {Aman Shenoy and Ashish Sardana},
  title = {{Multilogue-Net: A Context-Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure