TL;DR
This paper introduces DVL-CER, a novel dual-view learning framework for conversational emotion recognition (CER). By jointly modeling utterance context and emotion shifts, DVL-CER significantly improves emotion recognition accuracy, achieving state-of-the-art results on benchmark datasets.
What Problem Does This Paper Address?
Conversational Emotion Recognition (CER) aims to identify the emotions expressed in dialogue. While existing methods often focus on modeling the context between utterances, they tend to overlook the dynamics of a speaker's emotional state within a conversation. This "emotion-view" is crucial because emotions aren't static; they evolve. Current approaches often treat each utterance in isolation or only consider broad inter-utterance relationships, limiting their ability to capture subtle emotional nuances and transitions. This paper addresses this gap by proposing a method that explicitly models these emotion shifts, integrating this new perspective with traditional utterance-based context modeling.
Methods
The core of DVL-CER is its dual-view projection learning strategy. It comprises two main components:
- Conversational Context Network (Utterance-View): This is a standard CER model (e.g., RNN or GNN based) that processes utterances to learn their emotional distributions. This part focuses on the "what" of the current utterance within the dialogue flow.
- Emotion Shift Network (Emotion-View): This novel component uses a Transformer architecture to model how a speaker's emotions change over a sequence of utterances. It takes the predicted emotion distribution of an initial utterance and the subsequent utterances as input to predict the future emotion distribution. This captures the "how" and "why" of emotional transitions.
The key innovation lies in how these two views are integrated:
- Dual-View Projection Learning: Two projection heads are introduced. One head projects the emotion distribution from the utterance-view into the emotion-view's space, and the other projects from the emotion-view to the utterance-view. These projections are trained using Mean Squared Error (MSE) loss to encourage consistency between the two views. This allows the models to learn from each other, enhancing the robustness of the final emotion distributions.
- Emotion Validation Chain: The emotion shift network uses the predicted emotion distribution (rather than a hard label) as its initial input. This creates an implicit validation loop, where the model learns to predict and validate emotional transitions, leading to more accurate and nuanced emotion distributions.
Datasets
The proposed DVL-CER framework was evaluated on two widely-used benchmark datasets for conversational emotion recognition:
- IEMOCAP (Interactive Emotional Dyadic Motion Capture Database): Contains dyadic conversations with 6 emotion classes.
- EmoryNLP: Contains multi-party conversations with 7 emotion classes.
The paper uses only the textual modality from these datasets.
Key Insights and Ideas
- Emotion Shifts Matter: Explicitly modeling how a speaker's emotions evolve over time, not just their current state, significantly improves CER performance. This goes beyond simple utterance context.
- Distributions, Not Just Labels: Using predicted emotion distributions as input for emotion shift modeling, rather than hard emotion labels, provides a richer, more nuanced signal that leads to better performance. This creates an "emotion validation chain."
- Cross-View Projection for Synergy: Projecting emotion distributions between the utterance-view and emotion-view networks allows them to learn from each other, creating a synergistic effect that enhances the final emotion predictions. This is more effective than simply aggregating features from different views.
- Flexibility is Key: The DVL-CER framework is designed to be agnostic to the specific conversational context network used, allowing it to be easily integrated with various existing CER models.
Key Results
- State-of-the-Art Performance: DVL-CER achieved state-of-the-art results on both the IEMOCAP and EmoryNLP datasets, outperforming existing single-view and multi-view CER methods.
- Significant Improvements: On IEMOCAP, DVL-CER achieved 70.11% weighted-F1, an improvement of 1.10% over the best previous method (DialogueCRN+). On EmoryNLP, it reached 39.92% weighted-F1, an improvement of 0.95%.
- Effectiveness of Emotion Shift Modeling: Ablation studies confirmed that the emotion shift network is effective and contributes significantly to performance improvements.
- Superiority of Projection Learning: The proposed dual-view projection learning strategy outperformed other dual-view learning strategies (feature aggregation, subspace learning, alignment learning) in terms of balancing consistency and independence.
Limitations and Future Work
- Multimodal Extension: The authors suggest that future research could explore extending the emotion shift modeling to multimodal CER (e.g., incorporating audio and visual cues).
- Generalizability: While the framework is flexible, further exploration of its application to other multi-view supervised learning tasks is suggested.
- Emotion Shift Span: The optimal emotion shift span was found to be 4 utterances, but this might vary across different types of conversations or datasets. Further investigation into adaptive span selection could be beneficial.
Summary updated