Uncertain Graph Neural Networks for Facial Action Unit Detection

Tengfei Song, Lisha Chen, Wenming Zheng, Qiang JiAAAI 2021 · OpenReview

Abstract: Capturing the dependencies among different facial action units (AU) is extremely important for the AU detection task. Many studies have employed graph-based deep learning methods to exploit the dependencies among AUs. However, the dependencies among AUs in real world data are often noisy and the uncertainty is essential to be taken into consideration. Rather than employing a deterministic mode, we propose an uncertain graph neural network (UGN) to learn the probabilistic mask that simultaneously captures both the individual dependencies among AUs and the uncertainties. Further, we propose an adaptive weighted loss function based on the epistemic uncertainties to adaptively vary the weights of the training samples during the training process to account for unbalanced data distributions among AUs. We also provide an insightful analysis on how the uncertainties are related to the performance of AU detection. Extensive experiments, conducted on two benchmark datasets, i.e., BP4D and DISFA, demonstrate our method achieves the state-of-the-art performance.

Introduction

Facial Action Units (AUs) represent the fundamental muscle movements that constitute human expressions. Automatically detecting these units is a cornerstone of affective computing, yet it remains a significant challenge due to three primary factors. First, AUs exhibit complex dependencies; they rarely act in isolation, often appearing in specific semantic or anatomical combinations. Second, the ground truth labels are frequently noisy, as AU annotation requires high-level expertise and is prone to human error. Finally, facial datasets are often highly imbalanced, with common AUs like blinking appearing far more frequently than rare movements like nose wrinkling.

While existing graph-based methods have attempted to model AU dependencies, they typically rely on deterministic models that assume relationships are fixed and certain. This paper introduces the Uncertain Graph Neural Network (UGN), a framework designed to embrace and quantify uncertainty to build more robust detectors.

Methodology

The UGN framework moves away from static graph representations, instead modeling AU relationships as probabilistic distributions. The architecture consists of several key components:

Feature Extraction and Node Representation

The model utilizes a standard ResNet-18 backbone to extract visual features from facial images. These features are mapped to specific AU nodes, representing the initial state of the facial muscles.

The Probabilistic Mask

At the heart of the UGN is the probabilistic mask. Unlike traditional graph neural networks that use a fixed adjacency matrix, the UGN learns a distribution—defined by a mean (μ) and variance (σ)—for every pair of AUs. By sampling from this distribution using the reparameterization trick, the model generates a unique mask for each input image. This allows the network to account for individual differences in facial movement and the inherent noise in AU relationships.

The UGN architecture: (a) Feature extraction via ResNet and (b) the probabilistic mask generation and graph convolution process.

Modeling Uncertainty

The framework explicitly captures two types of uncertainty:

  1. Aleatoric Uncertainty: Represents noise inherent in the data itself, such as motion blur or ambiguous expressions.
  2. Epistemic Uncertainty: Represents the model’s lack of knowledge, typically occurring in scenarios with insufficient training data for specific AUs.

Adaptive Weighted Loss

To address data imbalance, the authors propose an adaptive weighted loss function. By calculating the epistemic uncertainty for each training sample, the model can identify "hard" or rare cases. Samples with higher uncertainty are automatically assigned higher weights in the loss function, forcing the model to prioritize learning from difficult and infrequent examples during the training process.

Experiments and Results

The UGN was evaluated on two gold-standard benchmarks: BP4D (approx. 140,000 frames) and DISFA (27 videos with intensity annotations).

Performance Metrics

The UGN achieved state-of-the-art (SOTA) results on both datasets. On BP4D, it reached an average F1-score of 63.3%, significantly outperforming the standard ResNet-18 baseline (58.7%) and deterministic graph models (62.6%). On the DISFA dataset, the improvement was even more pronounced, with the UGN achieving a 60.0% F1-score compared to the 48.5% typically seen in previous ROI-based methods.

Performance comparison between UGN and the deterministic GN-D baseline on (a) BP4D and (b) DISFA datasets across varying training data percentages.

As shown in the performance curves, the UGN consistently outperforms deterministic baselines (GN-D), particularly when training data is scarce. The model showed its most dramatic improvements on rare AUs (such as AU1, AU2, and AU23), validating the effectiveness of the adaptive weighted loss in countering data imbalance.

Discussion and Conclusion

A key insight from this research is that uncertainty should be treated as a valuable signal rather than a flaw. By identifying where the model is "confused," the UGN can dynamically rebalance the training process. Furthermore, the use of dynamic, image-specific graphs allows the model to adapt to the unique facial morphology of different individuals.

Despite its success, the model's sampling-based approach introduces higher computational overhead than deterministic models. Future work may explore incorporating temporal context—how AUs evolve over time—into the uncertain graph framework, as well as investigating self-supervised methods to further reduce the dependency on expert annotations. Ultimately, the UGN demonstrates that modeling the "unknowns" is a powerful strategy for mastering the complexities of facial expression analysis.

Summary updated

Abstract

Capturing the dependencies among different facial action units (AU) is extremely important for the AU detection task. Many studies have employed graph-based deep learning methods to exploit the dependencies among AUs. However, the dependencies among AUs in real world data are often noisy and the uncertainty is essential to be taken into consideration. Rather than employing a deterministic mode, we propose an uncertain graph neural network (UGN) to learn the probabilistic mask that simultaneously captures both the individual dependencies among AUs and the uncertainties. Further, we propose an adaptive weighted loss function based on the epistemic uncertainties to adaptively vary the weights of the training samples during the training process to account for unbalanced data distributions among AUs. We also provide an insightful analysis on how the uncertainties are related to the performance of AU detection. Extensive experiments, conducted on two benchmark datasets, i.e., BP4D and DISFA, demonstrate our method achieves the state-of-the-art performance.
BibTeX
@inproceedings{song2021uncertain,
  author = {Tengfei Song and Lisha Chen and Wenming Zheng and Qiang Ji},
  title = {{Uncertain Graph Neural Networks for Facial Action Unit Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=r34EdYPFpA4},
}
Figure