Video Representation Learning for Conversational Facial Expression Recognition Guided by Multiple View Reconstruction

Valeriya Strizhkova, Laura M. Ferrari, Hadi Kachmar, Antitza Dantcheva, François BrémondWorkshops @ CVPR 2024 · DOI

Abstract

Recognizing subtle facial expressions in conversational settings (cFER) is challenging due to speech interference, low intensity, and pose variations. We present MAURA (Masking Action Units and Reconstructing multiple Angles), a self-supervised pre-training method that leverages Action Unit (AU)-based masking and cross-view reconstruction. By forcing the model to reconstruct facial dynamics from unseen camera angles, MAURA learns robust 4D representations that achieve state-of-the-art performance on multiple challenging datasets, including "in-the-wild" scenarios.

Introduction

Conversational Facial Expression Recognition (cFER) is significantly more complex than standard facial recognition. The research identifies four primary hurdles:

  1. Speech Interference: Distinguishing facial movements caused by speech from those representing genuine emotion.
  2. Subtlety: Conversational expressions are often low-intensity or "fine-grained" (e.g., distinguishing "annoyed" from "furious").
  3. Pose Variations: Real-world interactions involve head movements that occlude parts of the face.
  4. Data Scarcity: High-quality, annotated video datasets for emotion are limited compared to general computer vision datasets.

Existing self-supervised methods like VideoMAE use random masking, which may inadvertently hide the critical muscle movements necessary for emotion recognition.

Methodology: The MAURA Framework

MAURA utilizes a Vision Transformer (ViT) architecture and introduces two novel strategies to overcome the limitations of standard masked autoencoding.

AU-Based Masking

Instead of random masking, the system employs the OpenFace library to detect Action Units (AUs)—specific muscle groups involved in expressions. MAURA ensures that video patches containing active AUs remain visible while masking the rest. This forces the encoder to focus on the relationship between muscle activations and facial geometry, rather than static background elements.

Cross-View Reconstruction

The "secret sauce" of MAURA is its reconstruction task. During pre-training, the model receives a video from one angle (e.g., frontal) but must reconstruct the video from a different angle (e.g., a 60° side view). This cross-view task is significantly harder than simple denoising; it requires the model to internalize the 3D structure and temporal dynamics of the face to "imagine" its appearance from a new perspective.

The MAURA framework: Pre-training involves AU-based masking and cross-view reconstruction, followed by fine-tuning for various FER tasks.
Detailed view of the MAURA pre-training process, showing the temporal reconstruction of a target view (V2) from a masked input view (V1).

Experimental Setup and Datasets

The model was evaluated across four diverse datasets:

  • MEAD: A large-scale multi-view dataset used for primary pre-training.
  • MFA: Focused on fine-grained negative emotions.
  • DFEW: A massive "in-the-wild" dataset from movies with extreme lighting and pose changes.
  • CMU-MOSEI: A benchmark for conversational analysis featuring 1,000 speakers.

Results and Key Insights

MAURA established new state-of-the-art (SOTA) benchmarks:

  • Low-Intensity Emotions: On MEAD (low), MAURA achieved a 51.9% F1-score, a significant improvement over the previous best of ~43%.
  • Fine-Grained Anger: On the MFA dataset, it reached a 55.6% F1-score.
  • In-the-Wild Performance: On the DFEW dataset, MAURA hit 47.5% F1-score, surpassing the previous SOTA of 43.6%.

The results suggest that harder pre-training tasks lead to better features. By requiring cross-view generation, the AI learns a "generic" representation of the face independent of camera perspective. Furthermore, the model's success on the DFEW dataset indicates that pre-training on lab-controlled data (MEAD) can effectively transfer to real-world scenarios. The model also proved highly effective for linear probing, indicating that the pre-trained features are robust enough for new tasks with minimal additional training.

Conclusion and Future Work

While MAURA demonstrates robust performance, it currently relies on OpenFace for AU detection, which can struggle at extreme angles. Future work aims to integrate multi-modal data (audio and text) to provide richer conversational context. The researchers also plan to explore applications in DeepFake detection and lip-synchronization.

Summary updated

BibTeX
@inproceedings{strizhkova2024video,
  author = {Valeriya Strizhkova and Laura M. Ferrari and Hadi Kachmar and Antitza Dantcheva and François Brémond},
  title = {{Video Representation Learning for Conversational Facial Expression Recognition Guided by Multiple View Reconstruction}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00472},
}
Figure