Introduction
Facial Action Units (AUs) represent the fundamental muscle movements that constitute human expressions. While detecting the presence of an AU is a well-studied task, estimating its intensity—typically on a 0–5 scale—remains a significant challenge in computer vision. This difficulty stems from the visual subtlety between adjacent intensity levels and the strong correlations between different AUs. For instance, a high-intensity smile (AU12) is frequently accompanied by a cheek raiser (AU6).
Methodology
The proposed framework addresses these challenges through a two-pronged approach: heatmap regression and Semantic Correspondence Convolution (SCC).
Heatmap Regression
Instead of traditional scalar regression, the model predicts a 2D heatmap for each AU. The peak value of the heatmap indicates the intensity, while its spatial location corresponds to the physical position of the AU on the face. This method preserves critical spatial context often lost in simpler models.
Semantic Correspondence Convolution (SCC)
The core innovation is the SCC module, which utilizes Dynamic Graph Convolutions to model relationships between facial features.
- Dynamic Graph Construction: The model treats feature channels as nodes and uses a k-Nearest Neighbors (k-NN) approach to identify semantically similar patterns.
- Edge Learning: Convolutions are performed across edges connecting related features, allowing the network to learn latent dependencies (e.g., the link between "lip stretching" and "mouth opening").
- Adaptive Refinement: The graph is updated dynamically during inference, enabling the model to adapt to individual facial variations rather than relying on static, predefined rules.
Experimental Results
The model was evaluated on the BP4D and DISFA datasets, both of which feature spontaneous expressions and imbalanced intensity distributions.
Performance was measured using the Intra-class Correlation (ICC) metric:
- BP4D: The model achieved an average ICC of 0.72, significantly outperforming prior state-of-the-art methods like CCNN-IT and KJRE (approx. 0.60–0.67).
- DISFA: It reached an average ICC of 0.47. Ablation studies confirmed the importance of the SCC module; removing it caused the ICC to drop to 0.40.
Conclusion
By modeling AU correlations at the feature level through dynamic graphs, this framework provides a robust solution for fine-grained facial analysis. Future work may explore more sophisticated graph construction methods and the application of this approach to unsupervised learning scenarios to reduce the reliance on expert-labeled data.
Summary updated