D-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection

Itir Önal Ertugrul, Le Yang, László A. Jeni, Jeffrey F. Cohn

Introduction

Facial Action Units (AUs) are the fundamental components of human expressions, representing specific muscle movements such as a "cheek raiser" or "lip corner puller." While automated AU detection has advanced significantly, researchers face three primary hurdles: pose sensitivity, rigid patch-to-AU mappings, and the difficulty of integrating temporal dynamics into feature learning. This paper introduces D-PAttNet (Dynamic Patch-Attentive Deep Network), a novel framework that addresses these challenges by integrating 3D face registration, joint static-dynamic patch encoding, and a unique sigmoidal attention mechanism.

Methodology

The D-PAttNet architecture mimics the hierarchical nature of human facial perception, processing information through four distinct stages.

The D-PAttNet architecture overview, illustrating the pipeline from 2D video to AU detection.

3D Face Registration

To mitigate pose sensitivity, the model utilizes 3D face registration to reconstruct a dense 3D mesh from 2D video. This process allows the system to normalize for pitch, yaw, and roll, ensuring that facial patches remain semantically consistent regardless of head movement.

Example of a 3D registered face used to ensure semantic consistency of patches.

Dual-Stream Patch Encoding

The model crops nine specific facial patches (e.g., brows, eyes, mouth). Each patch is fed into parallel encoders: a 2D-CNN for static frame-level information and a 3D-CNN to capture local spatiotemporal dynamics over a 20-frame window. By using 3D-CNNs at the patch level, the model learns local motion—such as the specific way a lip corner twitches—rather than just global sequence summaries.

Sigmoidal Attention Mechanism

A significant departure from traditional models is the use of a Sigmoid function for attention instead of the standard Softmax. Softmax attention is a "winner-take-all" approach that forces the model to select a single most important region. In contrast, Sigmoid attention allows the network to assign high importance to multiple patches independently, which is crucial because facial expressions often involve several muscle groups acting simultaneously.

AU-Specific Networks

Instead of a single multi-label classifier, the authors train individual networks for each AU. This design choice allows the model to better capture the specific regional nuances and muscle contractions associated with individual AUs.

Experimental Setup

The model was evaluated on the BP4D-Spontaneous dataset, comprising 328 videos from 41 subjects. The dataset features spontaneous expressions elicited through various tasks and manually annotated using the Facial Action Coding System (FACS). The study focused on 12 frequently occurring AUs.

Results and Discussion

D-PAttNet set a new performance benchmark for the BP4D dataset, achieving a 2.1% absolute increase in F1-score and a 0.7% increase in AUC compared to previous state-of-the-art models.

Ablation studies confirmed the value of the proposed components:

  • Dynamics: The inclusion of 3D-CNNs for temporal dynamics improved the average F1-score by 1.5% over a static-only version.
  • Attention: Switching from Softmax to Sigmoid attention resulted in a 2.4% increase in average F1-score, proving that "attending to many" is superior to "selecting one" for facial analysis.
  • 3D Normalization: The results suggest that 3D registration provides a cleaner signal for the deep network to learn subtle muscle contractions, even in datasets with limited head rotation.

Conclusion

D-PAttNet demonstrates that combining 3D registration with flexible attention and integrated dynamics significantly enhances AU detection. While the model shows promise, future work is needed to test its robustness in "in-the-wild" settings with extreme rotations and to extend the framework for estimating AU intensity rather than just occurrence.

Summary updated

Abstract

p>Facial action units (AUs) relate to specific local facial regions. Recent efforts in automated AU detection have focused on learning the facial patch representations to detect specific AUs. These efforts have encountered three hurdles. First, they implicitly assume that facial patches are robust to head rotation; yet non-frontal rotation is common. Second, mappings between AUs and patches are defined a priori, which ignores co-occurrences among AUs. And third, the dynamics of AUs are either ignored or modeled sequentially rather than simultaneously as in human perception. Inspired by recent advances in human perception, we propose a dynamic patch-attentive deep network, called D-PAttNet, for AU detection that (i) controls for 3D head and face rotation, (ii) learns mappings of patches to AUs, and (iii) models spatiotemporal dynamics. D-PAttNet approach significantly improves upon existing state of the art.</p>
BibTeX
@article{ertugrul2019d,
  author = {Itir Önal Ertugrul and Le Yang and László A. Jeni and Jeffrey F. Cohn},
  title = {{D-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection}},
  journal = {Frontiers of Computer Science},
  year = {2019},
  doi = {10.3389/fcomp.2019.00011},
}
Figure