EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration

Daiqing Wu, Dongbao Yang, Can Ma, Yu ZhouarXiv 2025 · DOI · arXiv

Introduction

Visual Emotion Comprehension (VEC) aims to enable artificial intelligence to interpret the affective content of images. While Multimodal Large Language Models (MLLMs) have demonstrated significant progress in this domain, they often fall into a "certainty trap"—producing deterministic labels for inherently subjective stimuli. This lack of transparency regarding internal uncertainty limits the utility of VEC in sensitive fields such as mental health and social robotics.

EmoCaliber addresses this by introducing a framework that teaches models to verbalize their confidence and calibrate it through reinforcement learning. By moving beyond simple label prediction, EmoCaliber provides a more nuanced, human-like understanding of visual emotion.

Methodology: The Three-Stage Evolution

The development of EmoCaliber follows a structured three-stage pipeline designed to transition the model from raw prediction to calibrated self-assessment.

The EmoCaliber training pipeline, transitioning from unbounded reasoning to structured reasoning and calibrated confidence.

Stage 1: Structured Reasoning (The Scaffold Model)

To prevent the model from jumping to conclusions, it is first trained on a structured "Chain-of-Thought" (CoT) reasoning process. This scaffold requires the model to analyze five key dimensions before predicting an emotion: visual elements, human subjects, environmental context, social interactions, and a final integrated analysis.

The five-step Chain-of-Thought (CoT) reasoning framework used to scaffold emotion analysis.

Stage 2: Confidence Verbalization (The Cold-Start Model)

In the second stage, the model learns to output a numerical confidence score alongside its prediction. To ensure these scores are meaningful, the researchers utilized the NRC VAD Lexicon (Valence, Arousal, Dominance). By measuring the semantic distance between predicted and ground-truth emotions in a psychological space, the model is taught that "near misses" (e.g., confusing "Amusement" with "Contentment") should correlate with higher confidence than "far misses" (e.g., confusing "Amusement" with "Anger").

Stage 3: Calibration via Reinforcement Learning (EmoCaliber)

The final stage employs Group Relative Policy Optimization (GRPO) to fine-tune the model. The reward function incentivizes correct formatting, accurate emotion prediction, and, crucially, calibrated confidence. The model is heavily penalized for being "confident but wrong," forcing it to align its verbalized certainty with its actual performance.

VECBench: A Unified Evaluation Framework

To rigorously test EmoCaliber, the authors introduced VECBench, a comprehensive benchmark consolidating six major datasets. This includes in-domain data like Flickr/Instagram (FI) and EmoSet, as well as challenging out-of-domain datasets like abstract art and unbiased object-based emotions. Additionally, the VEC-CoT dataset provides over 140,000 triplets of images, labels, and reasoning paths to support the scaffolding stage.

Key Results and Performance

EmoCaliber demonstrates state-of-the-art performance across multiple metrics:

  • Accuracy: It achieved an average accuracy of 53.70% on the ID VER benchmark, a significant improvement over previous models like Qwen3-VL.
  • Reliability: The Expected Calibration Error (ECE) was reduced to 13.63, nearly half the error rate of the next best open-source competitor (26.31). This indicates that EmoCaliber's self-reported confidence is a reliable indicator of its actual accuracy.
  • Generalization: The model maintained competitive performance even on out-of-domain tasks, rivaling proprietary systems like GPT-5 in its ability to handle abstract and artistic emotional content.

Conclusion

EmoCaliber represents a significant step toward more transparent and reliable visual emotion recognition. By integrating structured reasoning with RL-based confidence calibration, the framework allows MLLMs to acknowledge the subjectivity of emotion. Future work will focus on stabilizing the RL training process and extending these capabilities to temporal video-based emotion comprehension.

Summary updated

Abstract

Visual Emotion Comprehension (VEC) aims to infer sentiment polarities or emotion categories from affective cues embedded in images. In recent years, Multimodal Large Language Models (MLLMs) have established a popular paradigm in VEC, leveraging their generalizability to unify VEC tasks defined under diverse emotion taxonomies. While this paradigm achieves notable success, it typically formulates VEC as a deterministic task, requiring the model to output a single, definitive emotion label for each image. Such a formulation insufficiently accounts for the inherent subjectivity of emotion perception, overlooking alternative interpretations that may be equally plausible to different viewers. To address this limitation, we propose equipping MLLMs with capabilities to verbalize their confidence in emotion predictions. This additional signal provides users with an estimate of both the plausibility of alternative interpretations and the MLLMs' self-assessed competence, thereby enhancing reliability in practice. Building on this insight, we introduce a three-stage training framework that progressively endows with structured reasoning, teaches to verbalize confidence, and calibrates confidence expression, culminating in EmoCaliber, a confidence-aware MLLM for VEC. Through fair and comprehensive evaluations on the unified benchmark VECBench, EmoCaliber demonstrates overall superiority against existing methods in both emotion prediction and confidence estimation. These results validate the effectiveness of our approach and mark a feasible step toward more reliable VEC systems. Project page: https://github.com/wdqqdw/EmoCaliber.
BibTeX
@misc{wu2025emocaliber,
  author = {Daiqing Wu and Dongbao Yang and Can Ma and Yu Zhou},
  title = {{EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration}},
  year = {2025},
  eprint = {2512.15528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.15528},
}
Figure