G2RL: Geometry-Guided Representation Learning for Facial Action Unit Intensity Estimation

Yingruo Fan, Zhaojiang LinIJCAI 2020 · DOI

Introduction

Facial Action Units (AUs) are the fundamental building blocks of facial expressions, such as a "cheek raiser" or "lip corner puller." While recognizing the presence of an AU is a common task, estimating its intensity—typically on a 0-5 scale—presents a significantly greater challenge. This difficulty stems from individual physiological differences; for instance, a "lip corner puller" (AU12) manifests differently on a person with a wide jaw compared to someone with a narrower face.

Facial Action Unit intensity levels (0-5) across four different subjects, demonstrating the variability in expression magnitude.

Most existing models struggle to generalize across these varying facial geometries. While some researchers have attempted to use facial landmarks as auxiliary inputs, this often introduces significant computational overhead during deployment. The G2RL (Geometry-Guided Representation Learning) framework addresses this by "teaching" a visual model about face shape and muscle relationships during training, allowing it to internalize geometric constraints without requiring extra computation during inference.

Methodology

The G2RL framework is designed to bake "geometric wisdom" directly into a visual representation. It consists of three primary components:

  1. Heatmap Backbone: Utilizing a ResNet-50 architecture, the model generates heatmaps rather than simple scalar values. These heatmaps encode both the location and intensity of AUs, forcing the model to learn the spatial context of muscle movements.
  2. External Geometric Knowledge Module: A Graph Convolutional Network (GCN) processes 2D facial landmark coordinates. Since a face is structurally a graph of connected points, the GCN is uniquely suited to capture the spatial relationships and constraints between different facial regions.
  3. Knowledge Projection Layer: During training, the latent vector from the GCN is projected onto the visual backbone's feature maps. An auxiliary loss function encourages the backbone to align its features with this geometric information.
Visual examples of specific Action Units, such as AU10 (Upper Lip Raiser) and AU12 (Lip Corner Puller), which the model aims to quantify.

A key advantage of this approach is Zero-Cost Inference. Once training is complete, the GCN and projection layers are discarded. The backbone, having been "guided" by the geometric teacher, retains the learned spatial awareness, resulting in a lean and efficient model for real-world deployment.

Experimental Results

The model was evaluated on two major benchmarks: BP4D (328 videos, 41 subjects) and DISFA (27 videos, 27 subjects). Performance was measured using the Intraclass Correlation Coefficient (ICC) and Mean Absolute Error (MAE).

  • BP4D: G2RL achieved an average ICC of 0.69, surpassing previous state-of-the-art (SOTA) methods such as KBSS (0.67) and CCNN-IT (0.63).
  • DISFA: The model reached an average ICC of 0.52, a substantial improvement over the 0.40 baseline.

Ablation studies confirmed that the geometric guidance was the primary driver of these gains, as the baseline backbone without GCN-based supervision performed significantly worse.

Conclusion and Future Work

G2RL demonstrates that treating facial geometry as an external "teacher" can significantly enhance the precision of AU intensity estimation. By shifting from simple vector regression to heatmap-based learning and incorporating geometric constraints via auxiliary loss, the model achieves SOTA results.

However, challenges remain. Performance is currently better for frequently occurring AUs (like AU6 and AU12) than for rarer ones. Additionally, the model's training efficacy depends on the accuracy of the initial landmark detector. Future research may explore incorporating temporal dynamics—tracking how expressions evolve over time—to further refine intensity scores.

Summary updated

BibTeX
@inproceedings{fan2020g,
  author = {Yingruo Fan and Zhaojiang Lin},
  title = {{G2RL: Geometry-Guided Representation Learning for Facial Action Unit Intensity Estimation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/102},
}
Figure