TL;DR
This paper introduces a novel two-stage framework for multimodal emotion recognition in conversations (MERC). It uses a reinforcement learning approach (GRPO) to dynamically select which modalities (text, audio, visual) are most relevant for understanding emotion in a given utterance, preventing the model from being overwhelmed by redundant information. This adaptive selection, powered by LoRA-tuned LLMs, significantly improves emotion recognition accuracy on benchmark datasets.
What Problem Does This Paper Address?
Multimodal Emotion Recognition in Conversations (MERC) aims to understand a speaker's emotions by analyzing text, audio, and visual cues. While recent Large Language Models (LLMs) have shown promise, they often rely on a static approach, fusing all available modalities uniformly for every utterance. This is problematic because the importance of each modality can vary greatly. For instance, sometimes text alone is sufficient, while other times audio and visual cues are crucial to disambiguate meaning or provide context. Over-reliance on all modalities can introduce noise and dilute the model's focus. This paper addresses this limitation by proposing a method for adaptive modality selection, ensuring that only the necessary modalities are used for each specific conversational turn.
Methods
The proposed framework has two main stages:
- GRPO-Guided Modality Selection: This stage uses a LoRA-tuned LLM guided by Group Relative Policy Optimization (GRPO), a reinforcement learning technique. The LLM learns to reason (using Chain-of-Thought) about whether audio and visual descriptions are needed for the current utterance, considering the dialogue history. GRPO is chosen because it doesn't require explicit value functions or manually labeled modality selection data, making it suitable for unsupervised training. The model is prompted with the utterance, its history, and descriptions of the available modalities.
- LoRA-Tuned LLMs for Emotion Recognition: In the second stage, the selected modalities (text-only or multimodal) are used to construct a prompt for another LoRA-tuned LLM. This LLM then performs the final emotion classification.
Before these stages, raw audio and visual inputs are converted into structured textual descriptions using pre-trained models (Qwen2-Audio and Qwen2-VL) to ensure compatibility with LLMs.
Datasets
The paper evaluates its approach on two benchmark datasets:
- MELD: Contains 1,433 conversations and 13,708 utterances, labeled with seven emotions (neutral, surprise, fear, sadness, joy, disgust, anger).
- IEMOCAP: Contains 153 conversations and 7,433 utterances, labeled with six emotions (happy, sad, neutral, angry, excited, frustrated).
Key Insights and Ideas
- Modality necessity is dynamic: The core idea is that not all modalities are equally important for every utterance. Adaptive selection is key to robust emotion recognition.
- Unsupervised modality selection is feasible: The GRPO-based approach allows the model to learn when to use multimodal information without needing explicit labels for modality selection, significantly reducing annotation burden.
- LLMs can reason about modality relevance: By framing modality selection as a reasoning task, LLMs can effectively learn to identify when additional cues are beneficial.
- Over-reliance on modalities can be detrimental: The results show that simply including all modalities (as in VoiceERC) doesn't always improve performance and can even hurt it, especially when text is already informative. Adaptive selection offers a more nuanced approach.
- Contextual reasoning is crucial: The effectiveness of modality selection is highly dependent on the conversational context, highlighting the importance of historical dialogue information.
Key Results
The proposed GRPO-guided modality selection method consistently outperformed competitive baselines on both MELD and IEMOCAP datasets.
- It achieved the best overall performance, surpassing fine-tuned LLM baselines.
- Specifically, it improved the weighted F1 score by 1.65% on IEMOCAP and 0.94% on MELD compared to InstructERC.
- Ablation studies showed that removing GRPO led to a significant performance drop, confirming the critical role of dynamic modality selection.
- The adaptive strategy consistently outperformed a fixed integration baseline across different historical window lengths, with the gap widening in longer contexts.
- Fine-grained analysis showed the method achieved superior average performance and a more balanced distribution across emotion types on MELD.
Limitations and Future Work
The authors acknowledge two main limitations:
- Computational Constraints: The evaluation was limited to models with fewer than 10B parameters due to computational constraints.
- Pipeline Framework: The current method uses a pipeline approach, decoupling modality selection from emotion classification. While this offers modularity and interpretability, it might limit end-to-end optimization, which is an area for future exploration.
Summary updated