Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generation

Feifan Wang, Tengfei Song, Minggui He, Chang Su, Zhanglin Wu, Hao Yang, Wenming Zheng, Osamu YoshiearXiv 2025 · DOI · arXiv

Abstract

Vision Large Language Models (VLLMs) have demonstrated impressive general-purpose visual understanding but often fall short in fine-grained emotional intelligence. Tasks such as Action Unit (AU) detection and Valence-Arousal (VA) estimation remain challenging due to the scarcity of high-quality, multi-grained annotated data. We present SEKE (Self-verification with Emotion Knowledge Enhancement), a novel framework designed to generate high-quality emotion instruction data by leveraging teacher models, human prior knowledge, and uncertainty-aware sampling. Using SEKE, we constructed the FEID (Facial Emotion Instruction Dataset), which enables open-source models to surpass even closed-source giants like GPT-4o in complex emotional analysis tasks.

Introduction

Current VLLMs excel at coarse-grained emotion recognition (e.g., identifying a face as "happy") but struggle with the nuances required for deep emotional understanding. Specifically, they lack proficiency in identifying Action Units (AUs)—the fundamental muscle movements of facial expressions—and estimating Valence-Arousal (VA) values, which describe the positivity and intensity of an emotion. The primary obstacle is the lack of large-scale, expert-annotated datasets that link these different granularities, as high-quality emotion annotations typically require expensive psychological expertise.

Methodology: The SEKE Framework

The SEKE framework addresses the data bottleneck by combining two key components to generate reliable, multi-grained instructions:

  1. Emotion Knowledge Enhancement: Instead of prompting a model to guess labels in isolation, the framework provides "prior knowledge" (e.g., a known expression label) and leverages known psychological correlations between expressions, AUs, and VA values. This guides the model to infer missing fine-grained details within a constrained, logically consistent space.
  2. SV-UAMC (Uncertainty-Aware Monte Carlo Sampling): To mitigate hallucinations, the system employs Monte Carlo sampling and calculates epistemic uncertainty. If the model's outputs are inconsistent (high uncertainty), the system triggers additional sampling to reach a stable, verified consensus. This self-verification process filters out low-quality labels without human intervention.
The SEKE framework for high-quality emotion instruction data generation.

Datasets: FEID and FEAB

The authors introduce two major resources derived from aggregating and enriching six well-known datasets (CK+, RAF-DB, AffectNet, DISFA, BP4D, and Aff-Wild2):

  • FEID (Facial Emotion Instruction Dataset): A training set of 26,238 images with comprehensive multi-grained annotations (Expression, AU, and VA) and reasoning logic.
  • FEAB (Facial Emotion Analysis Benchmark): A test set of 3,035 images used to evaluate model performance across all three granularities.

Experimental Results

By fine-tuning a LLaVA-OneVision base model on the FEID dataset, the resulting SEKE model achieved state-of-the-art results on the FEAB benchmark, significantly outperforming both open-source and closed-source competitors.

Performance comparison across Expression Classification, AU Detection, and VA Estimation.

Key performance highlights include:

  • Expression Classification: Achieved 69.4% accuracy, surpassing GPT-4o's 61.1%.
  • Action Unit Detection: Reached an F1 score of 54.9%, nearly doubling GPT-4o’s 29.0%.
  • Valence/Arousal Estimation: Reduced Mean Absolute Error (MAE) to 0.295 for Valence, outperforming all other tested models.

Conclusion

The SEKE approach demonstrates that embedding psychological constraints and uncertainty-based self-verification into the data generation pipeline can significantly enhance the emotional intelligence of VLLMs. By training on multiple granularities simultaneously, the model develops a holistic understanding where different emotional descriptors reinforce one another. Future work will explore extending this framework to video-based analysis to capture temporal micro-expressions.

Summary updated

BibTeX
@misc{wang2025emotion,
  author = {Feifan Wang and Tengfei Song and Minggui He and Chang Su and Zhanglin Wu and Hao Yang and Wenming Zheng and Osamu Yoshie},
  title = {{Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generation}},
  year = {2025},
  eprint = {2505.18168},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.18168},
}
Figure