Trend-Aware Supervision: On Learning Invariance for Semi-Supervised Facial Action Unit Intensity Estimation

Yingjie Chen, Jiarui Zhang, Tao Wang, Yun Liang

Abstract

With the increasing need for facial behavior analysis, semi-supervised AU intensity estimation using only keyframe annotations has emerged as a practical and effective solution to relieve the burden of annotation. However, the lack of annotations makes the spurious correlation problem caused by AU co-occurrences and subject variation much more prominent, leading to non-robust intensity estimation that is entangled among AUs and biased among subjects. We observe that trend information inherent in keyframe annotations could act as extra supervision and raising the awareness of AU-specific facial appearance changing trends during training is the key to learning invariant AU-specific features. To this end, we propose Trend-Aware Supervision (TAS), which pursues three kinds of trend awareness, including intra-trend ranking awareness, intra-trend speed awareness, and inter-trend subject awareness. TAS alleviates the spurious correlation problem by raising trend awareness during training to learn AU-specific features that represent the corresponding facial appearance changes, to achieve intensity estimation invariance. Experiments conducted on two commonly used AU benchmark datasets, BP4D and DISFA, show the effectiveness of each kind of awareness. And under trend-aware supervision, the performance can be improved without extra computational or storage costs during inference.

Introduction and Problem Definition

Facial Action Units (AUs) are the fundamental components of facial expressions, such as a "cheek raiser" (AU6) or a "lip corner puller" (AU12). Accurately estimating their intensity is vital for applications ranging from medical pain assessment to driver fatigue monitoring. However, obtaining expert labels for every frame in a video sequence is prohibitively expensive and time-consuming.

Researchers often employ semi-supervised learning, utilizing only sparse "keyframes" (e.g., the start, peak, or end of a movement). A significant challenge in this low-data regime is the spurious correlation problem. Models often become "lazy," associating intensity with non-causal factors like a subject's unique bone structure (subject bias) or the tendency for certain muscles to activate simultaneously (co-occurrence bias). This leads to non-robust estimations that are entangled among AUs and biased among subjects.

Figure 1: Illustration of AU intensity trends, co-occurrence bias, and subject variation.

Methodology: Trend-Aware Supervision (TAS)

The core insight of this work is that the temporal flow between keyframes contains inherent "trend information" that can serve as additional supervision. By focusing on how expressions evolve over time, the model can learn to ignore spurious correlations and focus on AU-specific facial appearance changes. The proposed TAS framework consists of three primary components:

  1. Intra-Trend Ranking Awareness: Facial appearance should change monotonically between a "valley" (intensity 0) and a "peak" (intensity 5). TAS forces the model's internal latent features to follow this monotonic path, ensuring the feature representation itself reflects the progression of the expression.
  2. Intra-Trend Speed Awareness: To ensure smooth intensity estimation, a "mixup" technique is used to create virtual images by blending frames within a trend. The model is supervised to ensure that the intensity of these blended images matches a linear blend of the original intensities, helping the model learn the "speed" of expression changes.
  3. Inter-Trend Subject Awareness: To mitigate subject bias, TAS aligns features of the same AU at identical intensity levels across different individuals. This encourages the model to extract features common to the muscle movement while ignoring subject-specific facial characteristics.
Figure 2: The Trend-Aware Supervision (TAS) framework, highlighting the three awareness modules.

Experimental Evaluation

The TAS framework was evaluated on two gold-standard benchmarks:

  • BP4D: Spontaneous expressions containing ~140,000 frames from 41 subjects.
  • DISFA: Spontaneous expressions containing ~130,000 frames from 26 subjects.

Following a strict semi-supervised setup, only 1-2% of available annotations (the keyframes) were used for training. Despite this sparse supervision, the model achieved state-of-the-art results.

Key Results and Insights

  • Superior Accuracy: On the BP4D dataset, TAS achieved an average Intraclass Correlation (ICC) of 0.71, significantly outperforming previous semi-supervised methods like KBSS (0.65) and CFLF (0.66).
  • Competitive with Full Supervision: In several instances, TAS (using 2% labels) outperformed older models trained on 100% of the labels, demonstrating the power of trend-based feature learning.
  • Zero Inference Overhead: Since the trend-awareness modules are implemented as loss functions during the training phase, the final model remains as lightweight and fast as a standard ResNet-34 during inference.
  • Learning Invariance: By supervising the direction and speed of feature changes, the model successfully "strips away" non-causal features like gender or skin tone, leading to more robust AU-specific representations.

Conclusion and Future Work

Trend-Aware Supervision (TAS) effectively addresses the spurious correlation problem in semi-supervised AU intensity estimation by leveraging the temporal trends inherent in sparse keyframe data. While highly effective, the method currently depends on the accurate identification of keyframes. Future research may explore applying this trend-aware philosophy to other video-based regression tasks, such as gesture recognition or physical therapy monitoring, and testing robustness in "in-the-wild" scenarios involving extreme lighting or occlusions.

Summary updated

Abstract

With the increasing need for facial behavior analysis, semi-supervised AU intensity estimation using only keyframe annotations has emerged as a practical and effective solution to relieve the burden of annotation. However, the lack of annotations makes the spurious correlation problem caused by AU co-occurrences and subject variation much more prominent, leading to non-robust intensity estimation that is entangled among AUs and biased among subjects. We observe that trend information inherent in keyframe annotations could act as extra supervision and raising the awareness of AU-specific facial appearance changing trends during training is the key to learning invariant AU-specific features. To this end, we propose \textbf{T}rend-\textbf{A}ware \textbf{S}upervision (TAS), which pursues three kinds of trend awareness, including intra-trend ranking awareness, intra-trend speed awareness, and inter-trend subject awareness. TAS alleviates the spurious correlation problem by raising trend awareness during training to learn AU-specific features that represent the corresponding facial appearance changes, to achieve intensity estimation invariance. Experiments conducted on two commonly used AU benchmark datasets, BP4D and DISFA, show the effectiveness of each kind of awareness. And under trend-aware supervision, the performance can be improved without extra computational or storage costs during inference.
BibTeX
@inproceedings{chen2024trend,
  author = {Yingjie Chen and Jiarui Zhang and Tao Wang and Yun Liang},
  title = {{Trend-Aware Supervision: On Learning Invariance for Semi-Supervised Facial Action Unit Intensity Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27803},
}
Figure