Trend-Aware Supervision: On Learning Invariance for Semi-supervised Facial Action Unit Intensity Estimation

Yingjie Chen, Jiarui Zhang, Tao Wang, Yun Liang

Abstract

Facial Action Unit (AU) intensity estimation is a critical task in computer vision with applications ranging from healthcare to driver safety. However, the scarcity of expert-labeled data necessitates semi-supervised approaches, which are often plagued by spurious correlations such as AU co-occurrence and subject-specific biases. This paper introduces Trend-Aware Supervision (TAS), a method that utilizes the inherent temporal "flow" of facial expressions to supervise feature learning. TAS consists of three modules: Intra-Trend Ranking Awareness, Intra-Trend Speed Awareness, and Inter-Trend Subject Awareness. Experiments on BP4D and DISFA datasets demonstrate that TAS significantly outperforms existing semi-supervised methods and achieves performance comparable to fully supervised models using only 1-2% of labels.

Introduction

Facial Action Units (AUs) represent the fundamental muscular movements that constitute human expressions. Estimating their intensity—typically on a 0 to 5 scale—is essential for nuanced behavioral analysis, such as pain assessment or fatigue monitoring. A significant bottleneck in developing these models is the requirement for frame-by-frame expert annotation, which is both time-consuming and expensive.

To mitigate this, semi-supervised learning is often employed, typically utilizing only "keyframes" (peaks and valleys of expressions). However, this sparse labeling often leads models to learn spurious correlations. For instance, AU Co-occurrence might lead a model to incorrectly associate eye-squinting (AU6) features with smile (AU12) intensity. Similarly, Subject Variation can cause models to overfit to individual identities or demographic biases rather than the actual muscle movements.

Conceptual overview of temporal trends and the challenges of spurious correlations.

Methodology: Trend-Aware Supervision (TAS)

The core innovation of this work is Trend-Aware Supervision (TAS). Instead of treating video frames as isolated instances, TAS leverages the continuous temporal "trend" between a valley (neutral) and a peak (maximum intensity) to guide the learning process.

The Trend-Aware Supervision (TAS) framework architecture and its three core awareness components.

The TAS framework is built upon three primary components designed to learn invariant and causal features:

  1. Intra-Trend Ranking Awareness: This component enforces monotonicity. As a facial expression moves from a valley to a peak, the model's internal feature representations are constrained to change in a consistent, one-way direction, reflecting the physical progression of the muscle movement.
  2. Intra-Trend Speed Awareness: Utilizing a "Mixup" data augmentation strategy, the model creates virtual intermediate frames. It is then trained to ensure that its intensity predictions change linearly relative to these blends, effectively teaching the model the "speed" of the expression's change.
  3. Inter-Trend Subject Awareness: To decouple facial actions from individual identity, this module encourages the model to produce similar feature representations for the same AU intensity level across different subjects. This ensures that a "Level 3" intensity is recognized by the underlying muscle configuration, regardless of the person's gender or face shape.

Experimental Results

The TAS framework was evaluated on the BP4D and DISFA benchmarks, using only 2% and 1% of the available labels, respectively.

  • State-of-the-Art Performance: TAS outperformed existing semi-supervised methods. On the BP4D dataset, it achieved an Intraclass Correlation (ICC) of 0.71, surpassing the previous state-of-the-art (CFLF at 0.66).
  • Efficiency vs. Full Supervision: Remarkably, TAS achieved results comparable to models trained on 100% of the data. On DISFA, TAS's ICC of 0.52 outperformed several fully supervised models, including CCNN-IT (0.38) and standard CNNs (0.33).
  • Computational Efficiency: Since the trend-aware constraints are applied only during the training phase, the resulting inference model remains lightweight and efficient for real-time applications.

Conclusion

Trend-Aware Supervision offers a robust solution to the data scarcity problem in facial AU intensity estimation. By focusing on the temporal flow and structural information inherent in facial movements, TAS successfully disentangles facial actions from confounding factors like identity and co-occurring movements. Future work may extend this trend-based approach to other continuous video-based tasks, such as gesture recognition or physical therapy monitoring, and explore its robustness under extreme head poses and occlusions.

Summary updated

Abstract

With the increasing need for facial behavior analysis, semi-supervised AU intensity estimation using only keyframe annotations has emerged as a practical and effective solution to relieve the burden of annotation. However, the lack of annotations makes the spurious correlation problem caused by AU co-occurrences and subject variation much more prominent, leading to non-robust intensity estimation that is entangled among AUs and biased among subjects. We observe that trend information inherent in keyframe annotations could act as extra supervision and raising the awareness of AU-specific facial appearance changing trends during training is the key to learning invariant AU-specific features. To this end, we propose \textbf{T}rend-\textbf{A}ware \textbf{S}upervision (TAS), which pursues three kinds of trend awareness, including intra-trend ranking awareness, intra-trend speed awareness, and inter-trend subject awareness. TAS alleviates the spurious correlation problem by raising trend awareness during training to learn AU-specific features that represent the corresponding facial appearance changes, to achieve intensity estimation invariance. Experiments conducted on two commonly used AU benchmark datasets, BP4D and DISFA, show the effectiveness of each kind of awareness. And under trend-aware supervision, the performance can be improved without extra computational or storage costs during inference.
BibTeX
@inproceedings{chen2024trend,
  author = {Yingjie Chen and Jiarui Zhang and Tao Wang and Yun Liang},
  title = {{Trend-Aware Supervision: On Learning Invariance for Semi-supervised Facial Action Unit Intensity Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27803},
}
Figure