Abstract
Facial Action Unit (AU) intensity estimation is a critical task in computer vision with applications ranging from healthcare to driver safety. However, the scarcity of expert-labeled data necessitates semi-supervised approaches, which are often plagued by spurious correlations such as AU co-occurrence and subject-specific biases. This paper introduces Trend-Aware Supervision (TAS), a method that utilizes the inherent temporal "flow" of facial expressions to supervise feature learning. TAS consists of three modules: Intra-Trend Ranking Awareness, Intra-Trend Speed Awareness, and Inter-Trend Subject Awareness. Experiments on BP4D and DISFA datasets demonstrate that TAS significantly outperforms existing semi-supervised methods and achieves performance comparable to fully supervised models using only 1-2% of labels.
Introduction
Facial Action Units (AUs) represent the fundamental muscular movements that constitute human expressions. Estimating their intensity—typically on a 0 to 5 scale—is essential for nuanced behavioral analysis, such as pain assessment or fatigue monitoring. A significant bottleneck in developing these models is the requirement for frame-by-frame expert annotation, which is both time-consuming and expensive.
To mitigate this, semi-supervised learning is often employed, typically utilizing only "keyframes" (peaks and valleys of expressions). However, this sparse labeling often leads models to learn spurious correlations. For instance, AU Co-occurrence might lead a model to incorrectly associate eye-squinting (AU6) features with smile (AU12) intensity. Similarly, Subject Variation can cause models to overfit to individual identities or demographic biases rather than the actual muscle movements.
Methodology: Trend-Aware Supervision (TAS)
The core innovation of this work is Trend-Aware Supervision (TAS). Instead of treating video frames as isolated instances, TAS leverages the continuous temporal "trend" between a valley (neutral) and a peak (maximum intensity) to guide the learning process.
The TAS framework is built upon three primary components designed to learn invariant and causal features:
- Intra-Trend Ranking Awareness: This component enforces monotonicity. As a facial expression moves from a valley to a peak, the model's internal feature representations are constrained to change in a consistent, one-way direction, reflecting the physical progression of the muscle movement.
- Intra-Trend Speed Awareness: Utilizing a "Mixup" data augmentation strategy, the model creates virtual intermediate frames. It is then trained to ensure that its intensity predictions change linearly relative to these blends, effectively teaching the model the "speed" of the expression's change.
- Inter-Trend Subject Awareness: To decouple facial actions from individual identity, this module encourages the model to produce similar feature representations for the same AU intensity level across different subjects. This ensures that a "Level 3" intensity is recognized by the underlying muscle configuration, regardless of the person's gender or face shape.
Experimental Results
The TAS framework was evaluated on the BP4D and DISFA benchmarks, using only 2% and 1% of the available labels, respectively.
- State-of-the-Art Performance: TAS outperformed existing semi-supervised methods. On the BP4D dataset, it achieved an Intraclass Correlation (ICC) of 0.71, surpassing the previous state-of-the-art (CFLF at 0.66).
- Efficiency vs. Full Supervision: Remarkably, TAS achieved results comparable to models trained on 100% of the data. On DISFA, TAS's ICC of 0.52 outperformed several fully supervised models, including CCNN-IT (0.38) and standard CNNs (0.33).
- Computational Efficiency: Since the trend-aware constraints are applied only during the training phase, the resulting inference model remains lightweight and efficient for real-time applications.
Conclusion
Trend-Aware Supervision offers a robust solution to the data scarcity problem in facial AU intensity estimation. By focusing on the temporal flow and structural information inherent in facial movements, TAS successfully disentangles facial actions from confounding factors like identity and co-occurring movements. Future work may extend this trend-based approach to other continuous video-based tasks, such as gesture recognition or physical therapy monitoring, and explore its robustness under extreme head poses and occlusions.
Summary updated