Abstract
Facial Action Unit (FAU) detection is a cornerstone of affective computing, yet it faces significant challenges due to the scarcity of labeled video data and the difficulty of modeling complex temporal dynamics. We present AU-vMAE, a novel framework that combines self-supervised video pre-training with structured knowledge guidance. By utilizing a Video Masked Autoencoder (VideoMAE) to learn from massive unlabeled datasets and an AU-pair Finite State Machine (FSM) to model facial muscle transitions, AU-vMAE achieves state-of-the-art results on major benchmarks while maintaining high computational efficiency.
Introduction
Facial Action Units (FAUs) represent the fundamental muscle movements that constitute human expressions. Detecting these units is essential for psychological research and human-computer interaction. However, the field has been hindered by two primary obstacles. First, data scarcity: manually labeling every frame for dozens of AUs is labor-intensive, resulting in small datasets with limited diversity. Second, the static-dynamic gap: many existing methods treat facial expressions as static images, failing to capture the inherent temporal evolution and the physical constraints of how facial muscles interact over time.
Methodology
AU-vMAE addresses these challenges through a robust two-stage training pipeline designed to extract rich temporal features and enforce biological plausibility.
Self-Supervised Pre-training
To overcome data scarcity, the model first undergoes self-supervised pre-training using a Video Masked Autoencoder. The model is trained on approximately 0.5 million unlabeled videos. During this phase, a "tube masking" strategy is employed, where 90% of the visual data is masked. A Vision Transformer (ViT) encoder is then tasked with reconstructing the original video. This process forces the network to learn the underlying structural and temporal patterns of facial movements without requiring human-annotated labels.
Multi-Task Fine-tuning
Following pre-training, the model is fine-tuned using a multi-task classification approach. The framework operates across three levels simultaneously: video-level (processing all frames), frame-level (using downsampled video), and patch-level (handling masked frames). This multi-level strategy ensures that the feature extractor is robust to occlusions and varying temporal resolutions.
Knowledge-Guided AU Detection
A core innovation of AU-vMAE is the integration of the AU-pair Finite State Machine (FSM). Rather than attempting to model all AUs simultaneously—which leads to a state-space explosion—the authors focus on pairs of AUs to guide the network's learning.
- Intra-frame (Spatial) Correlation: The model learns the statistical likelihood of AUs occurring together within a single frame (e.g., the correlation between inner and outer brow raisers).
- Inter-frame (Temporal) Correlation: The FSM models the transition probabilities between frames. By treating AU pairs as a 4-bit state system (representing 16 possible transitions), the model ensures that predicted movements are physically and temporally consistent.
Experimental Results
AU-vMAE was evaluated on the BP4D and DISFA datasets, setting new performance benchmarks across the board:
- Performance Gains: The model achieved a 3.2% increase in F1-score on the BP4D dataset and a 5.8% increase on the DISFA dataset compared to previous state-of-the-art methods.
- Efficiency: Despite its high accuracy, the frame-level detection mode requires only 2.81 GFLOPs per frame. This is significantly more efficient than other Transformer-based models like ME-GraphAU, which requires 8.57 GFLOPs.
- Robustness: The model demonstrated high resilience to data loss, maintaining performance even when 50% of facial patches were randomly masked during inference.
Conclusion and Future Work
AU-vMAE demonstrates that combining large-scale self-supervised pre-training with structured prior knowledge is a powerful strategy for FAU detection. While the AU-pair approach effectively manages complexity, future research may explore more complex multi-AU dependencies and the model's performance in "in-the-wild" scenarios with extreme lighting and head poses. Addressing the inherent label imbalance in natural expression datasets also remains a critical area for further development.
Summary updated