Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning

Xuri Ge, Junchen Fu, Fuhai Chen, Shan An, Nicu Sebe, Joemon M. Jose

TL;DR

This paper introduces VL-FAU, a novel end-to-end system that not only recognizes facial action units (AUs) with state-of-the-art accuracy but also provides human-readable explanations for its predictions. By jointly learning vision and language tasks, VL-FAU enhances AU recognition by leveraging language as a form of semantic supervision.

What Problem Does This Paper Address?

Facial Action Unit (AU) recognition is crucial for understanding human emotions and expressions, with applications ranging from mental health diagnostics to human-computer interaction. However, existing methods primarily focus on predicting whether an AU is active or not, often treating it as a black box. This lack of explainability makes it difficult to trust or debug these systems. Furthermore, these models often struggle to differentiate between subtle variations in AUs or between different individuals, leading to performance limitations. The paper aims to bridge this gap by developing a system that is both accurate and interpretable, providing detailed descriptions of facial muscle movements alongside AU predictions.

Methods

The core of VL-FAU is a multi-branch network that leverages a pre-trained Swin-Transformer for initial facial feature extraction. The novelty lies in two key areas:

  • Dual-level AU Individual Refinement (DAIR): Each AU recognition branch is equipped with a DAIR module. This module uses multi-scale features and incorporates both channel-level and spatial-level attention mechanisms. This allows for more refined and attention-aware representations for each individual AU, improving their distinctiveness.
  • Vision-Language Joint Learning: This is the central innovation. VL-FAU integrates language generation modules that act as auxiliary supervisors.
  • Local Language Generation: For each AU branch, a dedicated language generator produces a fine-grained description of the specific muscle movements associated with that AU. This provides explicit semantic supervision, helping to disambiguate similar AUs and enrich their representations.
  • Global Language Generation: A separate language generator produces a holistic description of the entire facial expression, capturing the interplay of multiple AUs. This helps in distinguishing between different overall facial states, especially across different individuals and subtle variations.

These language generation tasks are trained jointly with the AU recognition task, using a combined loss function that includes AU classification loss, local language generation loss, and global language generation loss.

Datasets

The proposed VL-FAU model was evaluated on two widely used facial AU datasets:

  • DISFA: A spontaneous facial action intensity database.
  • BP4D: A spontaneous facial AU database.

The authors also mention that they hand-crafted language descriptions for different AU states for training these models.

Key Insights and Ideas

  • Language as Semantic Supervision: The paper demonstrates that language generation can serve as a powerful auxiliary task to improve the discriminative power of visual features for AU recognition. By forcing the model to describe what it "sees," it learns richer and more nuanced representations.
  • Local and Global Explanations Complement Each Other: Providing both fine-grained descriptions for individual AUs and a holistic description for the entire face leads to better overall performance and more comprehensive interpretability.
  • Improved AU Distinguishability: The joint learning approach, particularly with local language generation, helps in differentiating between AUs that might look visually similar but have distinct semantic meanings.
  • Enhanced Inter-Subject Discrimination: Global language generation, which incorporates gender information, aids in distinguishing between facial expressions of different individuals, even when they exhibit similar AU activations.

Key Results

  • State-of-the-Art Performance: VL-FAU achieved superior performance on both DISFA and BP4D datasets, outperforming existing state-of-the-art methods in terms of average F1-frame scores. For instance, on DISFA, it improved the average F1-frame score by 2.3% over the AAR method and from 64.5% to 66.5% compared to KDSRL. On BP4D, it also showed significant improvements.
  • Improved Accuracy: The model also achieved the best or second-best accuracy scores on both datasets compared to other methods.
  • Qualitative Improvements: Visualizations (t-SNE plots) showed that VL-FAU leads to better clustering and separation of AU features, as well as improved discrimination between subjects and genders.
  • Explainability: The model successfully generates both local (per-AU) and global (overall face) language descriptions that accurately reflect the predicted AU states, providing valuable interpretability.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the global description can sometimes ignore misrecognized AUs, the detailed local descriptions might still be affected by labeling ambiguity. The current approach relies on hand-crafted language descriptions for training.
  • Future Work: The authors suggest investigating the further combination of FAU recognition with popular Large Language Models (LLMs) to achieve more diverse and fine-grained explainable generations. They also highlight the potential for exploring more advanced multimodal joint training strategies.

Summary updated

Abstract

Facial action units (AUs), as defined in the Facial Action Coding System (FACS), have received significant research interest owing to their diverse range of applications in facial state analysis. Current mainstream FAU recognition models have a notable limitation, i.e., focusing only on the accuracy of AU recognition and overlooking explanations of corresponding AU states. In this paper, we propose an end-to-end Vision-Language joint learning network for explainable FAU recognition (termed VL-FAU), which aims to reinforce AU representation capability and language interpretability through the integration of joint multimodal tasks. Specifically, VL-FAU brings together language models to generate fine-grained local muscle descriptions and distinguishable global face description when optimising FAU recognition. Through this, the global facial representation and its local AU representations will achieve higher distinguishability among different AUs and different subjects. In addition, multi-level AU representation learning is utilised to improve AU individual attention-aware representation capabilities based on multi-scale combined facial stem feature. Extensive experiments on DISFA and BP4D AU datasets show that the proposed approach achieves superior performance over the state-of-the-art methods on most of the metrics. In addition, compared with mainstream FAU recognition methods, VL-FAU can provide local- and global-level interpretability language descriptions with the AUs' predictions.
BibTeX
@inproceedings{ge2024towards,
  author = {Xuri Ge and Junchen Fu and Fuhai Chen and Shan An and Nicu Sebe and Joemon M. Jose},
  title = {{Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681443},
}
Figure