Abstract: Capturing the dependencies among different facial action units (AU) is extremely important for the AU detection task. Many studies have employed graph-based deep learning methods to exploit the dependencies among AUs. However, the dependencies among AUs in real world data are often noisy and the uncertainty is essential to be taken into consideration. Rather than employing a deterministic mode, we propose an uncertain graph neural network (UGN) to learn the probabilistic mask that simultaneously captures both the individual dependencies among AUs and the uncertainties. Further, we propose an adaptive weighted loss function based on the epistemic uncertainties to adaptively vary the weights of the training samples during the training process to account for unbalanced data distributions among AUs. We also provide an insightful analysis on how the uncertainties are related to the performance of AU detection. Extensive experiments, conducted on two benchmark datasets, i.e., BP4D and DISFA, demonstrate our method achieves the state-of-the-art performance.
Introduction
Facial Action Units (AUs) represent the fundamental muscle movements that constitute human expressions. Automatically detecting these units is a cornerstone of affective computing, yet it remains a significant challenge due to three primary factors. First, AUs exhibit complex dependencies; they rarely act in isolation, often appearing in specific semantic or anatomical combinations. Second, the ground truth labels are frequently noisy, as AU annotation requires high-level expertise and is prone to human error. Finally, facial datasets are often highly imbalanced, with common AUs like blinking appearing far more frequently than rare movements like nose wrinkling.
While existing graph-based methods have attempted to model AU dependencies, they typically rely on deterministic models that assume relationships are fixed and certain. This paper introduces the Uncertain Graph Neural Network (UGN), a framework designed to embrace and quantify uncertainty to build more robust detectors.
Methodology
The UGN framework moves away from static graph representations, instead modeling AU relationships as probabilistic distributions. The architecture consists of several key components:
Feature Extraction and Node Representation
The model utilizes a standard ResNet-18 backbone to extract visual features from facial images. These features are mapped to specific AU nodes, representing the initial state of the facial muscles.
The Probabilistic Mask
At the heart of the UGN is the probabilistic mask. Unlike traditional graph neural networks that use a fixed adjacency matrix, the UGN learns a distribution—defined by a mean (μ) and variance (σ)—for every pair of AUs. By sampling from this distribution using the reparameterization trick, the model generates a unique mask for each input image. This allows the network to account for individual differences in facial movement and the inherent noise in AU relationships.
Modeling Uncertainty
The framework explicitly captures two types of uncertainty:
- Aleatoric Uncertainty: Represents noise inherent in the data itself, such as motion blur or ambiguous expressions.
- Epistemic Uncertainty: Represents the model’s lack of knowledge, typically occurring in scenarios with insufficient training data for specific AUs.
Adaptive Weighted Loss
To address data imbalance, the authors propose an adaptive weighted loss function. By calculating the epistemic uncertainty for each training sample, the model can identify "hard" or rare cases. Samples with higher uncertainty are automatically assigned higher weights in the loss function, forcing the model to prioritize learning from difficult and infrequent examples during the training process.
Experiments and Results
The UGN was evaluated on two gold-standard benchmarks: BP4D (approx. 140,000 frames) and DISFA (27 videos with intensity annotations).
Performance Metrics
The UGN achieved state-of-the-art (SOTA) results on both datasets. On BP4D, it reached an average F1-score of 63.3%, significantly outperforming the standard ResNet-18 baseline (58.7%) and deterministic graph models (62.6%). On the DISFA dataset, the improvement was even more pronounced, with the UGN achieving a 60.0% F1-score compared to the 48.5% typically seen in previous ROI-based methods.
As shown in the performance curves, the UGN consistently outperforms deterministic baselines (GN-D), particularly when training data is scarce. The model showed its most dramatic improvements on rare AUs (such as AU1, AU2, and AU23), validating the effectiveness of the adaptive weighted loss in countering data imbalance.
Discussion and Conclusion
A key insight from this research is that uncertainty should be treated as a valuable signal rather than a flaw. By identifying where the model is "confused," the UGN can dynamically rebalance the training process. Furthermore, the use of dynamic, image-specific graphs allows the model to adapt to the unique facial morphology of different individuals.
Despite its success, the model's sampling-based approach introduces higher computational overhead than deterministic models. Future work may explore incorporating temporal context—how AUs evolve over time—into the uncertain graph framework, as well as investigating self-supervised methods to further reduce the dependency on expert annotations. Ultimately, the UGN demonstrates that modeling the "unknowns" is a powerful strategy for mastering the complexities of facial expression analysis.
Summary updated