Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution

Yingruo Fan, Jacqueline C. K. Lam, Victor On Kwok LiAAAI 2020 · DOI

Introduction

Facial Action Units (AUs) represent the fundamental muscle movements that constitute human expressions. While detecting the presence of an AU is a well-studied task, estimating its intensity—typically on a 0–5 scale—remains a significant challenge in computer vision. This difficulty stems from the visual subtlety between adjacent intensity levels and the strong correlations between different AUs. For instance, a high-intensity smile (AU12) is frequently accompanied by a cheek raiser (AU6).

Visualizing AU correlations and intensity levels: (a) shows a high-intensity expression where AU6 and AU12 co-occur, while (b) shows a subtle, low-intensity AU12.

Methodology

The proposed framework addresses these challenges through a two-pronged approach: heatmap regression and Semantic Correspondence Convolution (SCC).

Heatmap Regression

Instead of traditional scalar regression, the model predicts a 2D heatmap for each AU. The peak value of the heatmap indicates the intensity, while its spatial location corresponds to the physical position of the AU on the face. This method preserves critical spatial context often lost in simpler models.

Semantic Correspondence Convolution (SCC)

The core innovation is the SCC module, which utilizes Dynamic Graph Convolutions to model relationships between facial features.

  • Dynamic Graph Construction: The model treats feature channels as nodes and uses a k-Nearest Neighbors (k-NN) approach to identify semantically similar patterns.
  • Edge Learning: Convolutions are performed across edges connecting related features, allowing the network to learn latent dependencies (e.g., the link between "lip stretching" and "mouth opening").
  • Adaptive Refinement: The graph is updated dynamically during inference, enabling the model to adapt to individual facial variations rather than relying on static, predefined rules.

Experimental Results

The model was evaluated on the BP4D and DISFA datasets, both of which feature spontaneous expressions and imbalanced intensity distributions.

Performance was measured using the Intra-class Correlation (ICC) metric:

  • BP4D: The model achieved an average ICC of 0.72, significantly outperforming prior state-of-the-art methods like CCNN-IT and KJRE (approx. 0.60–0.67).
  • DISFA: It reached an average ICC of 0.47. Ablation studies confirmed the importance of the SCC module; removing it caused the ICC to drop to 0.40.

Conclusion

By modeling AU correlations at the feature level through dynamic graphs, this framework provides a robust solution for fine-grained facial analysis. Future work may explore more sophisticated graph construction methods and the application of this approach to unsupervised learning scenarios to reduce the reliance on expert-labeled data.

Summary updated

BibTeX
@inproceedings{fan2020facial,
  author = {Yingruo Fan and Jacqueline C. K. Lam and Victor On Kwok Li},
  title = {{Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6963},
}
Figure