TL;DR
UEmo-D1 is the first "decision model" designed for unified Multimodal Emotion Recognition (MER), bridging the gap between efficient task-specific models and versatile but slow Multimodal Large Language Models (MLLMs). By mapping multimodal inputs directly to probability distributions over candidate options, it achieves state-of-the-art accuracy, lightning-fast inference, and human-like calibrated confidence across a wide array of emotional tasks.
What Problem Does This Paper Address?
Multimodal Emotion Recognition (MER) is the challenge of identifying human emotions from a mix of video, audio, and text. Historically, researchers have used two approaches:
- Task-specific models: These are fast and efficient but "brittle"—they don't generalize well to new types of data or different emotional labels.
- MLLM-based models: These use massive language models to "reason" about emotions. While they generalize well, they are computationally expensive, slow, and often "overconfident" (poorly calibrated), failing to reflect the ambiguity inherent in human emotions.
The authors argue that human emotion perception is often intuitive and rapid (akin to "System 1" thinking) and naturally accommodates multiple plausible interpretations. Existing models fail to reconcile three essential capabilities: cross-task generalization, efficient decision-making, and receptiveness to multiple interpretations.
Methods
The authors introduce UEmo-D1, a model based on the Qwen2.5-Omni-3B architecture, optimized to function as a "decision model." Instead of generating a long chain of reasoning, UEmo-D1 maps inputs directly to a probability distribution over a set of candidate options (e.g., A. Happy, B. Sad) at a single "decision-token" position.
Key technical innovations include:
- Bidirectional Attention: Replacing the standard causal (left-to-right) attention in the LLM backbone with BERT-style bidirectional attention. This allows every input token to "see" every other token, strengthening the contextual modeling across different modalities (vision, audio, text).
- Option-wise Regularization: A training modification that restricts the model's output normalization (softmax) only to the valid candidate letters (like A, B, or C). This focuses the model's learning specifically on the decision task.
- Temperature Scaling: A post-training calibration step that adjusts the model's confidence scores to ensure they accurately reflect the true probability of being correct.
Datasets
To train and evaluate such a versatile model, the authors curated UEmo-Data, a massive consolidated resource:
- Scale: Nearly 1 million training samples (977,270) and 41 evaluation subsets.
- Scope: It covers six distinct MER tasks:
- FER: Facial Expression Recognition.
- AER: Audio Emotion Recognition.
- VCEC: Visual Content Emotion Comprehension (how an image makes a viewer feel).
- CMER: Conversational Multimodal Emotion Recognition.
- ITSA: Image-Text Sentiment Analysis.
- MSU: Multimodal Social Understanding (detecting humor, sarcasm, or intent).
Key Insights and Ideas
- "Fast" can be "Better": Surprisingly, the "Decision Mode" (direct probabilistic output) often outperformed the "Reasoning Mode" (letting the model deliberate) in accuracy. This suggests that for emotion recognition, explicit step-by-step reasoning might sometimes introduce noise or overthinking.
- Calibration is Crucial: Most MLLMs are overconfident. UEmo-D1’s focus on calibration means that if it says there is a 70% chance of "Joy" and a 20% chance of "Surprise," those numbers are statistically reliable. This mirrors human-like receptiveness to multiple interpretations.
- Cross-Task Synergy: Training on a unified dataset allows the model to learn shared emotional representations. For instance, learning from facial expressions (FER) can improve the model's ability to understand social intent in videos (MSU).
Key Results
UEmo-D1 demonstrates a superior balance of accuracy, speed, and reliability compared to both open-source and proprietary giants like Gemini-3.5-Flash and GPT-4o-level models.
- Accuracy: Achieved an average accuracy of 75.6% across 41 subsets, outperforming the strongest baseline (Gemini-3.5-Flash) by 13.3%.
- Efficiency: Inference latency averaged just 0.55 seconds, significantly faster than models requiring multi-token reasoning.
- Calibration: Recorded an Expected Calibration Error (ECE) of 4.7%, which is roughly one-third of the error seen in the next-best model.
Limitations and Future Work
- Conflicting Modalities: The model still struggles when different modalities provide contradictory information (e.g., a person saying "I'm fine" with a sarcastic tone but a neutral face). The authors noted an "audio-dominant" bias in these cases.
- Asymmetric Transfer: While training on most tasks helped others, the relationship wasn't always perfectly reciprocal. Some tasks (like social understanding) seem to require higher-level integration that isn't fully captured by simpler tasks.
- Closed-set Constraints: Currently, the model operates on user-defined closed-set options. Extending this to open-vocabulary emotion recognition while maintaining calibration remains a challenge for future research.
Summary updated