AU-vMAE: Knowledge-Guide Action Units Detection via Video Masked Autoencoder

Qiaoqiao Jin, Rui Shi, Yishun Dou, Bingbing Ni

Abstract

Facial Action Unit (FAU) detection is a cornerstone of affective computing, yet it faces significant challenges due to the scarcity of labeled video data and the difficulty of modeling complex temporal dynamics. We present AU-vMAE, a novel framework that combines self-supervised video pre-training with structured knowledge guidance. By utilizing a Video Masked Autoencoder (VideoMAE) to learn from massive unlabeled datasets and an AU-pair Finite State Machine (FSM) to model facial muscle transitions, AU-vMAE achieves state-of-the-art results on major benchmarks while maintaining high computational efficiency.

Introduction

Facial Action Units (FAUs) represent the fundamental muscle movements that constitute human expressions. Detecting these units is essential for psychological research and human-computer interaction. However, the field has been hindered by two primary obstacles. First, data scarcity: manually labeling every frame for dozens of AUs is labor-intensive, resulting in small datasets with limited diversity. Second, the static-dynamic gap: many existing methods treat facial expressions as static images, failing to capture the inherent temporal evolution and the physical constraints of how facial muscles interact over time.

Methodology

AU-vMAE addresses these challenges through a robust two-stage training pipeline designed to extract rich temporal features and enforce biological plausibility.

Self-Supervised Pre-training

To overcome data scarcity, the model first undergoes self-supervised pre-training using a Video Masked Autoencoder. The model is trained on approximately 0.5 million unlabeled videos. During this phase, a "tube masking" strategy is employed, where 90% of the visual data is masked. A Vision Transformer (ViT) encoder is then tasked with reconstructing the original video. This process forces the network to learn the underlying structural and temporal patterns of facial movements without requiring human-annotated labels.

Multi-Task Fine-tuning

Following pre-training, the model is fine-tuned using a multi-task classification approach. The framework operates across three levels simultaneously: video-level (processing all frames), frame-level (using downsampled video), and patch-level (handling masked frames). This multi-level strategy ensures that the feature extractor is robust to occlusions and varying temporal resolutions.

The AU-vMAE architecture: (a) self-supervised pre-training via video reconstruction and (b) multi-task fine-tuning with knowledge guidance.

Knowledge-Guided AU Detection

A core innovation of AU-vMAE is the integration of the AU-pair Finite State Machine (FSM). Rather than attempting to model all AUs simultaneously—which leads to a state-space explosion—the authors focus on pairs of AUs to guide the network's learning.

  • Intra-frame (Spatial) Correlation: The model learns the statistical likelihood of AUs occurring together within a single frame (e.g., the correlation between inner and outer brow raisers).
  • Inter-frame (Temporal) Correlation: The FSM models the transition probabilities between frames. By treating AU pairs as a 4-bit state system (representing 16 possible transitions), the model ensures that predicted movements are physically and temporally consistent.
AU-pair Finite State Machine (FSM) illustrating transition probabilities for the 'Smile' expression (AU6 and AU12).

Experimental Results

AU-vMAE was evaluated on the BP4D and DISFA datasets, setting new performance benchmarks across the board:

  • Performance Gains: The model achieved a 3.2% increase in F1-score on the BP4D dataset and a 5.8% increase on the DISFA dataset compared to previous state-of-the-art methods.
  • Efficiency: Despite its high accuracy, the frame-level detection mode requires only 2.81 GFLOPs per frame. This is significantly more efficient than other Transformer-based models like ME-GraphAU, which requires 8.57 GFLOPs.
  • Robustness: The model demonstrated high resilience to data loss, maintaining performance even when 50% of facial patches were randomly masked during inference.

Conclusion and Future Work

AU-vMAE demonstrates that combining large-scale self-supervised pre-training with structured prior knowledge is a powerful strategy for FAU detection. While the AU-pair approach effectively manages complexity, future research may explore more complex multi-AU dependencies and the model's performance in "in-the-wild" scenarios with extreme lighting and head poses. Addressing the inherent label imbalance in natural expression datasets also remains a critical area for further development.

Summary updated

BibTeX
@inproceedings{jin2024au,
  author = {Qiaoqiao Jin and Rui Shi and Yishun Dou and Bingbing Ni},
  title = {{AU-vMAE: Knowledge-Guide Action Units Detection via Video Masked Autoencoder}},
  booktitle = {PRCV},
  year = {2024},
  doi = {10.1007/978-981-97-8499-8\_33},
}
Figure