Abstract
Emotion Recognition in Conversations (ERC) is a critical task for developing emotionally intelligent AI, yet it remains challenging due to the complex interplay of multimodal signals and conversational dynamics. Traditional models often rely on future information (future leakage) or fail to dynamically balance a speaker's internal emotional state with external influences. We present AdaIGN (Adaptive Interactive Graph Network), a framework that treats conversations as directed graphs and employs learnable selection policies to identify the most relevant modalities and contexts. AdaIGN effectively navigates the "inertia-empathy" tug-of-war, achieving state-of-the-art results on the IEMOCAP and MELD datasets.
The Challenge of Conversational Emotion Recognition
Human emotions in dialogue are rarely static; they are influenced by a combination of internal history and external social cues. The authors identify three primary hurdles in current ERC research:
- Future Leakage: Many models utilize "future" utterances to predict current emotions. While effective for offline analysis, this prevents real-time deployment where future data is unavailable.
- The Inertia-Empathy Tug-of-War: ERC requires a delicate balance between emotional inertia (the influence of a speaker’s own previous state) and empathy (the influence of the conversation partner). Most existing models treat these as static weights, failing to capture the fluid nature of human interaction.
- Multimodal Conflict: Modalities often provide conflicting signals—for instance, a speaker might use neutral words while exhibiting sad facial expressions. Standard fusion techniques often struggle to resolve these discrepancies.
As shown in Figure 1, a speaker's emotion (e.g., Rachel's sadness) can persist across multiple turns (inertia) or be triggered by the partner's inquiry (empathy).
The AdaIGN Framework
To address these challenges, the authors propose the Adaptive Interactive Graph Network (AdaIGN). This model represents a conversation as a directed graph where nodes correspond to multimodal utterances and edges represent the flow of influence over time.
Key Components:
- Directed Interactive Graph: By using directed edges, AdaIGN ensures that only past and current information informs the prediction, making the model strictly causal and suitable for real-time applications.
- Node and Edge-level Selection Policies (NESP): This serves as the core "decision-making" unit. Utilizing the Gumbel-Softmax trick, the model learns to dynamically "keep or drop" specific modalities (nodes) and historical influences (edges). This allows the model to prioritize the speaker's history or the partner's influence depending on the specific conversational context.
- Graph-Level Selection Policy (GSP): This component integrates the original graph features with the policy-refined features, ensuring a robust representation for final classification.
- Self-Supervised Learning: To improve efficiency, the model uses self-supervised pseudo-labels. When multiple modalities agree on an emotion, the model "masks" these easier cases, focusing its learning capacity on ambiguous or conflicting utterances.
Experimental Evaluation
The model was rigorously tested on two benchmark datasets:
- IEMOCAP: A dataset of professional actor performances (7,433 utterances) across six emotion categories.
- MELD: A multi-party dataset from the TV show Friends (13,708 utterances). MELD is notoriously difficult due to its noisy, real-world audio and complex multi-speaker dynamics.
Results and Insights
AdaIGN established new state-of-the-art (SOTA) benchmarks:
- IEMOCAP: Achieved a weighted F1-score of 70.74%, surpassing previous models like CMCF-SRNet.
- MELD: Achieved a weighted F1-score of 66.79%, representing a significant 4.49% improvement over previous SOTA. This highlights AdaIGN's robustness in noisy, multi-party environments.
Ablation studies confirmed that the selection policies (NSP/ESP) are vital; removing them led to a measurable performance decline. This proves that filtering information is as critical as processing it.
Conclusion and Future Directions
AdaIGN demonstrates that adaptive, directed graph learning is a powerful approach for real-time ERC. By dynamically balancing emotional inertia and empathy, it more closely mirrors human social cognition. However, challenges remain, particularly regarding class imbalance (e.g., low performance on rare emotions like "fear") and sudden emotional shifts. Future work will likely focus on improving the model's sensitivity to these abrupt transitions and leveraging more advanced self-supervised pre-training techniques.
Summary updated