Prompting Visual-Language Models for Dynamic Facial Expression Recognition

Zengqun Zhao, Ioannis Patras

Abstract

Dynamic Facial Expression Recognition (DFER) in "in-the-wild" scenarios remains challenging due to environmental variability and the subtle nature of temporal facial movements. This paper introduces DFER-CLIP, a framework that leverages the pre-trained CLIP model for video-based emotion recognition. By integrating a temporal Transformer to capture motion and utilizing fine-grained, LLM-generated descriptions of facial muscle movements, DFER-CLIP bridges the temporal and granularity gaps inherent in standard vision-language models. The approach achieves state-of-the-art performance on major DFER benchmarks, including DFEW, FERV39k, and MAFW.

Introduction

Recognizing emotions in real-world settings is significantly more complex than in controlled laboratory environments. "In-the-wild" DFER must account for varying lighting, head poses, and occlusions. While Vision-Language Pre-training (V-LP) models like CLIP have set new standards in image understanding, their application to DFER is hindered by two primary issues:

  1. The Temporal Gap: Standard CLIP models are designed for static images and lack an inherent mechanism to process temporal dynamics, which are essential for distinguishing between similar facial expressions.
  2. The Granularity Gap: CLIP is typically prompted with broad class names (e.g., "happiness"). However, accurately identifying subtle expressions requires understanding specific facial "mechanics," such as eye crinkling or cheek lifting.

Methodology: The DFER-CLIP Framework

DFER-CLIP addresses these challenges through a dual-stream architecture comprising a visual stream and a textual stream.

The detailed architecture of DFER-CLIP, illustrating the visual stream with a temporal model and the textual stream using LLM-based descriptors.

Visual Stream and Temporal Modeling

The visual stream processes a sequence of video frames through a shared CLIP image encoder. To bridge the temporal gap, a Temporal Model—consisting of shallow Transformer encoder layers—is added to learn relationships between frames. A learnable "class token" aggregates these temporal features into a unified video-level representation.

Textual Stream and Fine-Grained Descriptors

Instead of relying on simple emotion labels, the authors utilized ChatGPT to generate detailed descriptions of facial behaviors (Action Units) associated with each emotion. For instance, "sadness" is described by "drooping upper eyelids and a wrinkled forehead." Furthermore, learnable context tokens are introduced to optimize the prompts automatically during training, moving beyond fixed templates like "a photo of...".

Experimental Evaluation

The model was evaluated on three prominent "in-the-wild" benchmarks: DFEW (11,697 clips), FERV39k (38,935 clips), and MAFW (10,045 clips).

Key Results

DFER-CLIP established new state-of-the-art (SOTA) results across all tested datasets. As shown in the comparison below, the inclusion of the Temporal Model (TM) and descriptive prompts significantly outperforms both traditional classifier-based methods and standard zero-shot CLIP approaches.

Performance comparison of DFER-CLIP against various classifier-based and text-based strategies across DFEW, FERV39k, and MAFW datasets.

On the DFEW dataset, DFER-CLIP achieved a Weighted Average Recall (WAR) of 71.25%. It also reached 51.65% and 52.55% WAR on FERV39k and MAFW, respectively. Ablation studies confirmed that using physical facial descriptions is more effective than simple emotion labels, as it directs the model's attention to relevant muscle movements.

Discussion and Conclusion

The success of DFER-CLIP highlights the importance of aligning video features with nuanced textual descriptions in a shared embedding space. Interestingly, shallow temporal models often outperformed deeper ones, likely due to the risk of overfitting on relatively small DFER datasets.

Future work may address current limitations, such as the sensitivity to the number of learnable context tokens and the computational cost of fine-tuning large vision-language models on video data. Exploring "prompt ensembling" with multiple descriptions per class also presents a promising avenue for capturing the full diversity of emotional expressions.

Summary updated

BibTeX
@inproceedings{zhao2023prompting,
  author = {Zengqun Zhao and Ioannis Patras},
  title = {{Prompting Visual-Language Models for Dynamic Facial Expression Recognition}},
  booktitle = {BMVC},
  year = {2023},
  url = {https://openreview.net/forum?id=hWewGAV0WM},
}
Figure