Abstract
Facial Action Unit (AU) detection in unconstrained real-world environments is hindered by the lack of expert-annotated labels. This paper introduces ADLD (AU Detection via Latent Domain), a framework that leverages feature disentanglement and domain adaptation. By separating landmark-related (expression) features from landmark-free (pose/lighting) features and swapping them between constrained lab datasets and unconstrained wild datasets, the model learns robust AU detectors without requiring manual labels for target images.
Introduction
Facial Action Unit (AU) detection—identifying specific muscle movements like a brow raiser or a lip corner puller—is a cornerstone of affective computing. While AI performs well on faces captured in controlled laboratory settings, it struggles "in the wild" where lighting, head poses, and occlusions vary significantly.
The core challenge is the domain gap between clean, labeled lab data and messy, unlabeled real-world data. Manually annotating thousands of real-world images for AUs is labor-intensive and requires certified experts. ADLD addresses this by transferring knowledge from labeled lab data to unlabeled wild data through a novel latent feature approach.
Methods
The authors propose an end-to-end framework based on domain adaptation and feature disentanglement. The process follows a "Disentangle-Swap-Translate" pipeline:
- Disentanglement: A feature encoder ($E_f$) extracts "rich features" from both source (lab) and target (wild) images. These are split into landmark-related features ($z_l$), which capture AU-specific muscle movements, and landmark-free features ($z_t$), which capture global pose, lighting, and background.
- Landmark Adversarial Loss ($L_{adl}$): To ensure the landmark-free features are truly "clean," a discriminator tries to predict landmarks from these features, while the encoder is trained to make that impossible via a multi-player minimax game.
- The Swap: To train a detector for the "wild" domain, the model combines the landmark-related features from the lab (which have accurate AU labels) with the landmark-free features from the wild.
- Latent Feature Domain: This hybrid feature is mapped into a "latent domain" where an AU detector ($F_a$) is trained. Because this domain contains real-world noise but lab-accurate expression signals, the detector learns to be robust to unconstrained environments.
Datasets
The study utilized a comprehensive suite of benchmarks to validate the approach:
- Source Domains (Constrained): BP4D, GFT, DISFA, and UNBC-McMaster Shoulder Pain.
- Target Domains (Unconstrained): EmotioNet (1 million internet images) and VGGFace2.
- Annotations: While target images lacked AU labels, the authors used OpenPose to generate facial landmark labels, which serve as the anchor for transferring labels across domains.
Key Insights
- Landmarks as a Bridge: While AU labels are hard to obtain for real-world images, facial landmarks are relatively easy to track. Since AUs are physically tied to landmarks, they serve as a reliable proxy for feature transfer.
- Feature Swapping vs. Style Transfer: Unlike methods that attempt to make lab images look like real-world images, ADLD operates in the feature space. This avoids the computational cost and artifacts associated with high-resolution image generation.
- Multi-Player Minimax: Treating landmark disentanglement as a multi-class classification problem within a multi-player game achieved more stable and effective feature separation than standard two-player adversarial training.
Results
The ADLD framework consistently outperformed both "lower-bound" models (trained only on source data) and "upper-bound" models (using pseudo-labels).
- On the EmotioNet benchmark, ADLD achieved an average F1-score of 36.8, an 11.3 point margin over the basic domain adaptation baseline.
- When extended with pseudo-labels (ADLD-Full), the performance reached 42.4, surpassing state-of-the-art methods like JAA-Net and ARL.
- The model proved highly robust; even when trained on one source and tested on a completely different target, it maintained superior accuracy.
Conclusion and Future Work
While ADLD represents a significant step forward, the authors acknowledge that perfect disentanglement remains difficult. Some AU-related information may remain in the landmark-free features, or pose information may leak into the expression features. Future work will focus on refining the disentanglement process to handle extreme poses where landmarks are occluded and exploring applications in other computer vision tasks where labeled data is scarce.
Summary updated