UEmo-D1: An Efficient and Calibrated Decision Model for Unified Multimodal Emotion Recognition

Daiqing Wu, Junqiao Wang, Chengyang Fang, Runhao Zeng, Fei Ma, Sicheng Zhao

TL;DR

UEmo-D1 is the first "decision model" designed for unified Multimodal Emotion Recognition (MER), bridging the gap between efficient task-specific models and versatile but slow Multimodal Large Language Models (MLLMs). By mapping multimodal inputs directly to probability distributions over candidate options, it achieves state-of-the-art accuracy, lightning-fast inference, and human-like calibrated confidence across a wide array of emotional tasks.

What Problem Does This Paper Address?

Multimodal Emotion Recognition (MER) is the challenge of identifying human emotions from a mix of video, audio, and text. Historically, researchers have used two approaches:

  1. Task-specific models: These are fast and efficient but "brittle"—they don't generalize well to new types of data or different emotional labels.
  2. MLLM-based models: These use massive language models to "reason" about emotions. While they generalize well, they are computationally expensive, slow, and often "overconfident" (poorly calibrated), failing to reflect the ambiguity inherent in human emotions.

The authors argue that human emotion perception is often intuitive and rapid (akin to "System 1" thinking) and naturally accommodates multiple plausible interpretations. Existing models fail to reconcile three essential capabilities: cross-task generalization, efficient decision-making, and receptiveness to multiple interpretations.

Methods

The authors introduce UEmo-D1, a model based on the Qwen2.5-Omni-3B architecture, optimized to function as a "decision model." Instead of generating a long chain of reasoning, UEmo-D1 maps inputs directly to a probability distribution over a set of candidate options (e.g., A. Happy, B. Sad) at a single "decision-token" position.

Key technical innovations include:

  • Bidirectional Attention: Replacing the standard causal (left-to-right) attention in the LLM backbone with BERT-style bidirectional attention. This allows every input token to "see" every other token, strengthening the contextual modeling across different modalities (vision, audio, text).
  • Option-wise Regularization: A training modification that restricts the model's output normalization (softmax) only to the valid candidate letters (like A, B, or C). This focuses the model's learning specifically on the decision task.
  • Temperature Scaling: A post-training calibration step that adjusts the model's confidence scores to ensure they accurately reflect the true probability of being correct.
Figure 4. Overview of the training and calibration pipeline for UEmo-D1. Starting from the UEmo-Data train split, samples with different modality combinations undergo unchanged modality-specific encoding before entering the language backbone, where causal attention is replaced with bidirectional attention to strengthen contextual modeling. At the output decision-token position, option-wise regularization then restricts softmax normalization to valid options for cross-entropy supervision. Throughout training, only LoRA parameters [23] in the language backbone are updated, with all original model parameters frozen. After training, softmax temperature scaling is fitted on the validation split to yield the calibrated UEmo-D1.

Datasets

To train and evaluate such a versatile model, the authors curated UEmo-Data, a massive consolidated resource:

  • Scale: Nearly 1 million training samples (977,270) and 41 evaluation subsets.
  • Scope: It covers six distinct MER tasks:
  • FER: Facial Expression Recognition.
  • AER: Audio Emotion Recognition.
  • VCEC: Visual Content Emotion Comprehension (how an image makes a viewer feel).
  • CMER: Conversational Multimodal Emotion Recognition.
  • ITSA: Image-Text Sentiment Analysis.
  • MSU: Multimodal Social Understanding (detecting humor, sarcasm, or intent).
Figure 2. Overview of the UEmo-Data construction pipeline. Heterogeneous datasets from diverse sources spanning six MER tasks are collected and processed through five stages: label mapping, deduplication, quality control, split verification, and instruction formatting. The processed data are then consolidated into UEmo-Data, a unified resource for MER training and evaluation.

Key Insights and Ideas

  • "Fast" can be "Better": Surprisingly, the "Decision Mode" (direct probabilistic output) often outperformed the "Reasoning Mode" (letting the model deliberate) in accuracy. This suggests that for emotion recognition, explicit step-by-step reasoning might sometimes introduce noise or overthinking.
  • Calibration is Crucial: Most MLLMs are overconfident. UEmo-D1’s focus on calibration means that if it says there is a 70% chance of "Joy" and a 20% chance of "Surprise," those numbers are statistically reliable. This mirrors human-like receptiveness to multiple interpretations.
  • Cross-Task Synergy: Training on a unified dataset allows the model to learn shared emotional representations. For instance, learning from facial expressions (FER) can improve the model's ability to understand social intent in videos (MSU).

Key Results

UEmo-D1 demonstrates a superior balance of accuracy, speed, and reliability compared to both open-source and proprietary giants like Gemini-3.5-Flash and GPT-4o-level models.

  • Accuracy: Achieved an average accuracy of 75.6% across 41 subsets, outperforming the strongest baseline (Gemini-3.5-Flash) by 13.3%.
  • Efficiency: Inference latency averaged just 0.55 seconds, significantly faster than models requiring multi-token reasoning.
  • Calibration: Recorded an Expected Calibration Error (ECE) of 4.7%, which is roughly one-third of the error seen in the next-best model.
Figure 1. Illustration of UEmo-D1’s decision-making framework and comparison of its accuracy, efficiency, and calibration with ad- vanced MLLMs, with performance averaged over 41 subsets span- ning six MER tasks. Decision mode compares first-token option probabilities, whereas reasoning mode allows for deliberation. In- ference latency is measured on two NVIDIA RTX 4080 GPUs.

Limitations and Future Work

  • Conflicting Modalities: The model still struggles when different modalities provide contradictory information (e.g., a person saying "I'm fine" with a sarcastic tone but a neutral face). The authors noted an "audio-dominant" bias in these cases.
  • Asymmetric Transfer: While training on most tasks helped others, the relationship wasn't always perfectly reciprocal. Some tasks (like social understanding) seem to require higher-level integration that isn't fully captured by simpler tasks.
  • Closed-set Constraints: Currently, the model operates on user-defined closed-set options. Extending this to open-vocabulary emotion recognition while maintaining calibration remains a challenge for future research.

Summary updated

Abstract

Multimodal Emotion Recognition (MER) seeks to identify emotions conveyed through or elicited by media comprising diverse modality compositions. Research in this field has largely evolved along two paradigms: task-specific models and approaches based on Multimodal Large Language Models (MLLMs). Despite their distinct strengths, neither fully reconciles three capabilities essential to human-like emotion understanding: cross-task generalization, efficient decision-making, and receptiveness to multiple plausible interpretations. Inspired by recent advances in decision models, we explore a unified framework that maps multimodal inputs, guided by natural language instructions, directly to distributions over candidate options, addressing these requirements through direct probabilistic decisions. We instantiate this framework in UEmo-D1, the first decision model for MER to our knowledge. To support joint training and systematic evaluation, we also curate UEmo-Data, comprising 977,270 training samples and 41 evaluation subsets across six MER tasks. Extensive experiments demonstrate UEmo-D1's combination of strong recognition performance, low inference latency, and well-calibrated confidence estimates. Complementary analyses further reveal how training strategies and data composition influence these capabilities, providing practical insights into unified MER modeling. Together, our findings establish decision models as a feasible direction toward human-level emotion recognition. Project page: https://github.com/wdqqdw/UEmo-D1.
BibTeX
@misc{wu2026uemo,
  author = {Daiqing Wu and Junqiao Wang and Chengyang Fang and Runhao Zeng and Fei Ma and Sicheng Zhao},
  title = {{UEmo-D1: An Efficient and Calibrated Decision Model for Unified Multimodal Emotion Recognition}},
  year = {2026},
  eprint = {2611.24481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24481},
}
Figure