TL;DR
This paper introduces VL-FAU, a novel end-to-end system that not only recognizes facial action units (AUs) with state-of-the-art accuracy but also provides human-readable explanations for its predictions. By jointly learning vision and language tasks, VL-FAU enhances AU recognition by leveraging language as a form of semantic supervision.
What Problem Does This Paper Address?
Facial Action Unit (AU) recognition is crucial for understanding human emotions and expressions, with applications ranging from mental health diagnostics to human-computer interaction. However, existing methods primarily focus on predicting whether an AU is active or not, often treating it as a black box. This lack of explainability makes it difficult to trust or debug these systems. Furthermore, these models often struggle to differentiate between subtle variations in AUs or between different individuals, leading to performance limitations. The paper aims to bridge this gap by developing a system that is both accurate and interpretable, providing detailed descriptions of facial muscle movements alongside AU predictions.
Methods
The core of VL-FAU is a multi-branch network that leverages a pre-trained Swin-Transformer for initial facial feature extraction. The novelty lies in two key areas:
- Dual-level AU Individual Refinement (DAIR): Each AU recognition branch is equipped with a DAIR module. This module uses multi-scale features and incorporates both channel-level and spatial-level attention mechanisms. This allows for more refined and attention-aware representations for each individual AU, improving their distinctiveness.
- Vision-Language Joint Learning: This is the central innovation. VL-FAU integrates language generation modules that act as auxiliary supervisors.
- Local Language Generation: For each AU branch, a dedicated language generator produces a fine-grained description of the specific muscle movements associated with that AU. This provides explicit semantic supervision, helping to disambiguate similar AUs and enrich their representations.
- Global Language Generation: A separate language generator produces a holistic description of the entire facial expression, capturing the interplay of multiple AUs. This helps in distinguishing between different overall facial states, especially across different individuals and subtle variations.
These language generation tasks are trained jointly with the AU recognition task, using a combined loss function that includes AU classification loss, local language generation loss, and global language generation loss.
Datasets
The proposed VL-FAU model was evaluated on two widely used facial AU datasets:
- DISFA: A spontaneous facial action intensity database.
- BP4D: A spontaneous facial AU database.
The authors also mention that they hand-crafted language descriptions for different AU states for training these models.
Key Insights and Ideas
- Language as Semantic Supervision: The paper demonstrates that language generation can serve as a powerful auxiliary task to improve the discriminative power of visual features for AU recognition. By forcing the model to describe what it "sees," it learns richer and more nuanced representations.
- Local and Global Explanations Complement Each Other: Providing both fine-grained descriptions for individual AUs and a holistic description for the entire face leads to better overall performance and more comprehensive interpretability.
- Improved AU Distinguishability: The joint learning approach, particularly with local language generation, helps in differentiating between AUs that might look visually similar but have distinct semantic meanings.
- Enhanced Inter-Subject Discrimination: Global language generation, which incorporates gender information, aids in distinguishing between facial expressions of different individuals, even when they exhibit similar AU activations.
Key Results
- State-of-the-Art Performance: VL-FAU achieved superior performance on both DISFA and BP4D datasets, outperforming existing state-of-the-art methods in terms of average F1-frame scores. For instance, on DISFA, it improved the average F1-frame score by 2.3% over the AAR method and from 64.5% to 66.5% compared to KDSRL. On BP4D, it also showed significant improvements.
- Improved Accuracy: The model also achieved the best or second-best accuracy scores on both datasets compared to other methods.
- Qualitative Improvements: Visualizations (t-SNE plots) showed that VL-FAU leads to better clustering and separation of AU features, as well as improved discrimination between subjects and genders.
- Explainability: The model successfully generates both local (per-AU) and global (overall face) language descriptions that accurately reflect the predicted AU states, providing valuable interpretability.
Limitations and Future Work
- Limitations: The paper acknowledges that while the global description can sometimes ignore misrecognized AUs, the detailed local descriptions might still be affected by labeling ambiguity. The current approach relies on hand-crafted language descriptions for training.
- Future Work: The authors suggest investigating the further combination of FAU recognition with popular Large Language Models (LLMs) to achieve more diverse and fine-grained explainable generations. They also highlight the potential for exploring more advanced multimodal joint training strategies.
Summary updated