Exploiting Semantic Embedding and Visual Feature for Facial Action Unit Detection

Huiyuan Yang, Lijun Yin, Yi Zhou, Jiuxiang GuCVPR 2021 · OpenReview

Abstract

Recent study on detecting facial action units (AU) has utilized auxiliary information (i.e., facial landmarks, relationship among AUs and expressions, web facial images, etc.), in order to improve the AU detection performance. As of now, no semantic information of AUs has yet been explored for such a task. As a matter of fact, AU semantic descriptions provide much more information than the binary AU labels alone, thus we propose to exploit the Semantic Embedding and Visual feature (SEV-Net) for AU detection. More specifically, AU semantic embeddings are obtained through both Intra-AU and Inter-AU attention modules, where the Intra-AU attention module captures the relation among words within each sentence that describes individual AU, and the Inter-AU attention module focuses on the relation among those sentences. The learned AU semantic embeddings are then used as guidance for the generation of attention maps through a cross-modality attention network. The generated cross-modality attention maps are further used as weights for the aggregated feature. Our proposed method is unique in that the semantic features are exploited as the first of this kind. The approach has been evaluated on three public AU-coded facial expression databases, and has achieved a superior performance than the state-of-the-art peer methods.

Introduction

Facial Action Unit (AU) detection—the identification of specific muscle movements underlying facial expressions—is a fundamental task in affective computing. Despite its importance, the field faces significant challenges, primarily data scarcity due to the high cost of expert labeling and the underutilization of existing domain knowledge. Most current models treat AU labels as simple binary indicators, ignoring the rich, descriptive information available in the Facial Action Coding System (FACS). FACS provides detailed textual definitions of where actions occur and how the skin and muscles move. While some researchers have attempted to model AU relationships using manually constructed graphs, these are often rigid and biased. This paper introduces SEV-Net, a framework that leverages the actual text of FACS descriptions to guide visual feature extraction.

Methodology: The SEV-Net Framework

The SEV-Net (Semantic Embedding and Visual feature Network) architecture integrates visual information with semantic domain knowledge through two parallel tracks.

Visual and Semantic Tracks

The Visual Track employs a ResNet-18 backbone to extract spatial features from input face images. Simultaneously, the Semantic Track processes AU descriptions using a hierarchical attention mechanism:

  1. Intra-AU Attention: A Transformer-based encoder analyzes words within individual AU descriptions (e.g., "wrinkle the nose") to capture the specific location and nature of the action.
  2. Inter-AU Attention: This module models the relationships between different AU descriptions, automatically learning dependencies and co-occurrence patterns without the need for manual graph construction.

Cross-Modality Attention

The core of SEV-Net is the Cross-Modality Attention module, which uses the learned semantic embeddings to filter and weight the visual features. By acting as a spatial spotlight, the text embeddings direct the model's focus toward the facial regions most relevant to the target AUs.

The SEV-Net architecture integrates image encoding with hierarchical semantic encoding of AU descriptions to generate cross-modality attention maps.

Experimental Results

The performance of SEV-Net was evaluated on three benchmark datasets: BP4D, DISFA, and BP4D+.

Quantitative Performance

SEV-Net consistently outperformed state-of-the-art (SOTA) methods across all benchmarks. On the BP4D dataset, it achieved an average F1-score of 63.9%, a notable improvement over the ResNet-18 baseline. Similar gains were observed on DISFA (58.8% F1-score) and BP4D+ (61.5% F1-score), where it established new performance standards.

F1-score comparison across BP4D, DISFA, and BP4D+ datasets, demonstrating SEV-Net's superior performance over baseline and SOTA methods.

Qualitative Insights

Beyond quantitative metrics, visualization of the cross-modality attention maps revealed that SEV-Net accurately targets specific facial muscle groups described in the FACS manual. This interpretability distinguishes SEV-Net from traditional "black box" convolutional neural networks, providing visual evidence for its detection decisions.

Conclusion

SEV-Net demonstrates that semantic descriptions are a valuable and previously untapped source of domain knowledge for AU detection. By automatically learning AU relationships and using text to guide visual attention, the model achieves superior accuracy and interpretability. Future research will focus on integrating temporal dynamics to capture AUs defined by movement over time and exploring the model's sensitivity to linguistic variations in AU descriptions.

Summary updated

Abstract

Recent study on detecting facial action units (AU) has utilized auxiliary information (i.e., facial landmarks, relationship among AUs and expressions, web facial images, etc.), in order to improve the AU detection performance. As of now, no semantic information of AUs has yet been explored for such a task. As a matter of fact, AU semantic descriptions provide much more information than the binary AU labels alone, thus we propose to exploit the Semantic Embedding and Visual feature (SEV-Net) for AU detection. More specifically, AU semantic embeddings are obtained through both Intra-AU and Inter-AU attention modules, where the Intra-AU attention module captures the relation among words within each sentence that describes individual AU, and the Inter-AU attention module focuses on the relation among those sentences. The learned AU semantic embeddings are then used as guidance for the generation of attention maps through a cross-modality attention network. The generated cross-modality attention maps are further used as weights for the aggregated feature. Our proposed method is unique in that the semantic features are exploited as the first of this kind. The approach has been evaluated on three public AU-coded facial expression databases, and has achieved a superior performance than the state-of-the-art peer methods.
BibTeX
@inproceedings{yang2021exploiting,
  author = {Huiyuan Yang and Lijun Yin and Yi Zhou and Jiuxiang Gu},
  title = {{Exploiting Semantic Embedding and Visual Feature for Facial Action Unit Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=K4v4aYKB_Q},
}
Figure