Introduction
Facial Action Units (AUs) are the fundamental building blocks of facial expressions, such as a "cheek raiser" or "lip corner puller." While recognizing the presence of an AU is a common task, estimating its intensity—typically on a 0-5 scale—presents a significantly greater challenge. This difficulty stems from individual physiological differences; for instance, a "lip corner puller" (AU12) manifests differently on a person with a wide jaw compared to someone with a narrower face.
Most existing models struggle to generalize across these varying facial geometries. While some researchers have attempted to use facial landmarks as auxiliary inputs, this often introduces significant computational overhead during deployment. The G2RL (Geometry-Guided Representation Learning) framework addresses this by "teaching" a visual model about face shape and muscle relationships during training, allowing it to internalize geometric constraints without requiring extra computation during inference.
Methodology
The G2RL framework is designed to bake "geometric wisdom" directly into a visual representation. It consists of three primary components:
- Heatmap Backbone: Utilizing a ResNet-50 architecture, the model generates heatmaps rather than simple scalar values. These heatmaps encode both the location and intensity of AUs, forcing the model to learn the spatial context of muscle movements.
- External Geometric Knowledge Module: A Graph Convolutional Network (GCN) processes 2D facial landmark coordinates. Since a face is structurally a graph of connected points, the GCN is uniquely suited to capture the spatial relationships and constraints between different facial regions.
- Knowledge Projection Layer: During training, the latent vector from the GCN is projected onto the visual backbone's feature maps. An auxiliary loss function encourages the backbone to align its features with this geometric information.
A key advantage of this approach is Zero-Cost Inference. Once training is complete, the GCN and projection layers are discarded. The backbone, having been "guided" by the geometric teacher, retains the learned spatial awareness, resulting in a lean and efficient model for real-world deployment.
Experimental Results
The model was evaluated on two major benchmarks: BP4D (328 videos, 41 subjects) and DISFA (27 videos, 27 subjects). Performance was measured using the Intraclass Correlation Coefficient (ICC) and Mean Absolute Error (MAE).
- BP4D: G2RL achieved an average ICC of 0.69, surpassing previous state-of-the-art (SOTA) methods such as KBSS (0.67) and CCNN-IT (0.63).
- DISFA: The model reached an average ICC of 0.52, a substantial improvement over the 0.40 baseline.
Ablation studies confirmed that the geometric guidance was the primary driver of these gains, as the baseline backbone without GCN-based supervision performed significantly worse.
Conclusion and Future Work
G2RL demonstrates that treating facial geometry as an external "teacher" can significantly enhance the precision of AU intensity estimation. By shifting from simple vector regression to heatmap-based learning and incorporating geometric constraints via auxiliary loss, the model achieves SOTA results.
However, challenges remain. Performance is currently better for frequently occurring AUs (like AU6 and AU12) than for rarer ones. Additionally, the model's training efficacy depends on the accuracy of the initial landmark detector. Future research may explore incorporating temporal dynamics—tracking how expressions evolve over time—to further refine intensity scores.
Summary updated