Weak to Strong: VLM-Based Pseudo-Labeling as a Weakly Supervised Training Strategy in Multimodal Video-based Hidden Emotion Understanding Tasks

Yufei Wang, Haixu Liu, Tianxiang Xu, Chuancheng Shi, Hongsheng Xing

TL;DR

This paper introduces a novel multimodal framework for recognizing "hidden emotions" in videos, which are subtle and often concealed. They leverage a Vision-Language Model (VLM) to generate pseudo-labels for weakly supervised training, achieving state-of-the-art results on the iMiGUE dataset and addressing the challenge of limited labeled data.

What Problem Does This Paper Address?

Recognizing "hidden emotions" in videos is a challenging task. Unlike overt emotional expressions, it requires detecting subtle cues like micro-expressions, micro-movements, and intonation changes, often requiring multimodal analysis (visual, auditory, and textual). Existing methods struggle with data scarcity, labeling difficulties, weak and sparse signals, and the complexity of modeling long video sequences. This paper specifically targets the problem of accurately identifying these concealed emotions, which has significant implications for psychological assessment, clinical intervention, and security applications.

Methods

The proposed method employs a multimodal approach, integrating visual features from human portraits, human pose keypoints, and textual transcriptions.

  • Visual Feature Extraction: YOLO11x is used to detect and crop human figures frame-by-frame. Dinov2-Base then extracts visual features from these cropped regions, creating an image feature time series.
  • Pose Feature Extraction: OpenPose is utilized to extract 137 keypoints (body, face, hands). These keypoints are augmented with inter-frame offset features and normalized coordinates. Instead of traditional Graph Neural Networks (GCNs), the authors found that a simplified Multi-Layer Perceptron (MLP) combined with a Transformer effectively models the spatiotemporal relationships of these keypoints.
  • Textual Feature Extraction: BERT is used to encode the interview transcripts.
  • Weak Supervision with VLM: A key innovation is the use of Gemini 2.5 Pro with Chain-of-Thought and Reflection prompting (CoT + Reflection). This VLM generates pseudo-labels and reasoning texts by analyzing action units (FACS, PGCS, GACS) and inferring win/loss probabilities, serving as weak supervision for downstream models.
  • Multimodal Fusion: Features from the image, keypoint, and text modalities are independently encoded using Transformers and then concatenated and fused via a fully connected layer.

The training strategy involves a two-stage process: first, modality-specific pre-training, and then joint fine-tuning using a combination of the original training data and the VLM-generated pseudo-labeled test data.

Datasets

The primary dataset used is iMiGUE [1]. This dataset consists of 359 post-match press conference videos from the Australian Open, featuring 72 tennis players from 28 countries. The videos are muted, and emotions are indirectly labeled based on match results (win/loss). A significant challenge with this dataset is its class imbalance, with positive (win) samples accounting for approximately 75% of the data. The authors also generated pseudo-labels for the test set using Gemini 2.5 Pro.

Key Insights and Ideas

  • VLM for Weak Supervision is Powerful: The paper demonstrates that large Vision-Language Models like Gemini 2.5 Pro, when guided by carefully designed prompts (CoT + Reflection), can effectively generate useful pseudo-labels for a challenging task like hidden emotion recognition, even with a significant class imbalance.
  • MLP Outperforms GCN for Keypoint Modeling: Surprisingly, a simpler MLP-based backbone for keypoint feature extraction, when combined with a Transformer, achieved comparable or even better results than more complex GCN-based approaches. This suggests that explicit graph structures might not be as crucial as initially thought for modeling spatiotemporal keypoint relationships in this context, potentially due to the limitations of predefined graph priors in capturing complex micro-movements.
  • Dense Frame Sampling is Crucial: For visual features, dense frame sampling and modeling long temporal sequences with Transformers proved more effective than methods that sample fewer frames (like X-CLIP), allowing for the capture of fleeting micro-expressions.
  • Modality-Specific Pretraining Boosts Performance: Pretraining each modality independently before joint fine-tuning significantly improves overall model accuracy.

Key Results

  • The proposed multimodal framework achieved state-of-the-art accuracy on the iMiGUE dataset, improving accuracy from under 0.6 in prior work to over 0.69.
  • The MLP-based keypoint feature extraction method matched or surpassed GCN-based counterparts.
  • The VLM-based pseudo-labeling strategy was crucial for improving performance, especially given the data scarcity and class imbalance.
  • Modality-specific pretraining followed by overall weakly supervised fine-tuning yielded the greatest improvement in model accuracy.

Limitations and Future Work

Limitations:

  • Inaccurate Hand Keypoints: The quality of hand keypoint detection from OpenPose is poor, introducing significant noise into the hand modality.
  • Lack of Theoretical Framework for Noisy Labels: The paper acknowledges the absence of a formal theoretical understanding of how their weak supervision strategy generalizes with noisy labels and at what noise threshold performance degrades.
  • Unstable Gains from Semi-Supervised Learning: Attempts to use semi-supervised learning for further refinement did not consistently yield improvements, possibly due to issues with high-confidence sample filtering.
  • Limited Finetuning: Due to data scarcity, they could not extensively finetune large models like DINOv2 or more powerful text encoders, which may have capped overall performance.
  • Hyperparameter Selection: Hyperparameters were largely selected via leaderboard feedback, assuming no distribution shift between training and test sets, which is a questionable premise.

Future Work:

  • Leveraging 4D Facial Analysis: Incorporating 4D facial analysis techniques (e.g., GAGAvatar) to capture temporal evolution of facial expressions in 3D could enhance microexpression recognition.
  • Binding 2D Keypoints to Semantic Patches: Establishing a correspondence between 2D keypoints and visual features from image patches to create a more discriminative, sparse, structured feature set for temporal modeling.

Summary updated

BibTeX
@inproceedings{wang2025weak,
  author = {Yufei Wang and Haixu Liu and Tianxiang Xu and Chuancheng Shi and Hongsheng Xing},
  title = {{Weak to Strong: VLM-Based Pseudo-Labeling as a Weakly Supervised Training Strategy in Multimodal Video-based Hidden Emotion Understanding Tasks}},
  booktitle = {MiGA @ IJCAI},
  year = {2025},
}
Figure