Abstract
Recent study on detecting facial action units (AU) has utilized auxiliary information (i.e., facial landmarks, relationship among AUs and expressions, web facial images, etc.), in order to improve the AU detection performance. As of now, no semantic information of AUs has yet been explored for such a task. As a matter of fact, AU semantic descriptions provide much more information than the binary AU labels alone, thus we propose to exploit the Semantic Embedding and Visual feature (SEV-Net) for AU detection. More specifically, AU semantic embeddings are obtained through both Intra-AU and Inter-AU attention modules, where the Intra-AU attention module captures the relation among words within each sentence that describes individual AU, and the Inter-AU attention module focuses on the relation among those sentences. The learned AU semantic embeddings are then used as guidance for the generation of attention maps through a cross-modality attention network. The generated cross-modality attention maps are further used as weights for the aggregated feature. Our proposed method is unique in that the semantic features are exploited as the first of this kind. The approach has been evaluated on three public AU-coded facial expression databases, and has achieved a superior performance than the state-of-the-art peer methods.
Introduction
Facial Action Unit (AU) detection—the identification of specific muscle movements underlying facial expressions—is a fundamental task in affective computing. Despite its importance, the field faces significant challenges, primarily data scarcity due to the high cost of expert labeling and the underutilization of existing domain knowledge. Most current models treat AU labels as simple binary indicators, ignoring the rich, descriptive information available in the Facial Action Coding System (FACS). FACS provides detailed textual definitions of where actions occur and how the skin and muscles move. While some researchers have attempted to model AU relationships using manually constructed graphs, these are often rigid and biased. This paper introduces SEV-Net, a framework that leverages the actual text of FACS descriptions to guide visual feature extraction.
Methodology: The SEV-Net Framework
The SEV-Net (Semantic Embedding and Visual feature Network) architecture integrates visual information with semantic domain knowledge through two parallel tracks.
Visual and Semantic Tracks
The Visual Track employs a ResNet-18 backbone to extract spatial features from input face images. Simultaneously, the Semantic Track processes AU descriptions using a hierarchical attention mechanism:
- Intra-AU Attention: A Transformer-based encoder analyzes words within individual AU descriptions (e.g., "wrinkle the nose") to capture the specific location and nature of the action.
- Inter-AU Attention: This module models the relationships between different AU descriptions, automatically learning dependencies and co-occurrence patterns without the need for manual graph construction.
Cross-Modality Attention
The core of SEV-Net is the Cross-Modality Attention module, which uses the learned semantic embeddings to filter and weight the visual features. By acting as a spatial spotlight, the text embeddings direct the model's focus toward the facial regions most relevant to the target AUs.
Experimental Results
The performance of SEV-Net was evaluated on three benchmark datasets: BP4D, DISFA, and BP4D+.
Quantitative Performance
SEV-Net consistently outperformed state-of-the-art (SOTA) methods across all benchmarks. On the BP4D dataset, it achieved an average F1-score of 63.9%, a notable improvement over the ResNet-18 baseline. Similar gains were observed on DISFA (58.8% F1-score) and BP4D+ (61.5% F1-score), where it established new performance standards.
Qualitative Insights
Beyond quantitative metrics, visualization of the cross-modality attention maps revealed that SEV-Net accurately targets specific facial muscle groups described in the FACS manual. This interpretability distinguishes SEV-Net from traditional "black box" convolutional neural networks, providing visual evidence for its detection decisions.
Conclusion
SEV-Net demonstrates that semantic descriptions are a valuable and previously untapped source of domain knowledge for AU detection. By automatically learning AU relationships and using text to guide visual attention, the model achieves superior accuracy and interpretability. Future research will focus on integrating temporal dynamics to capture AUs defined by movement over time and exploring the model's sensitivity to linguistic variations in AU descriptions.
Summary updated