Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues

Soufiane Belharbi, Marco Pedersoli, Alessandro Lameiras Koerich, Simon Bacon, Eric Granger

Executive Summary

Recent advancements in deep learning have pushed Facial Expression Recognition (FER) to high levels of accuracy. However, these models often operate as "black boxes," making decisions based on spurious correlations rather than biologically relevant facial features. This research introduces a novel training strategy that enforces "intrinsic interpretability" by aligning a model's internal attention with expert-defined facial Action Units (AUs). By guiding the model to "look" at the specific muscle movements associated with emotions, the researchers achieve high interpretability without sacrificing classification performance.

The Challenge: Beyond Black-Box Predictions

In high-stakes domains such as e-health, driver monitoring, or psychological assessment, a simple probability score (e.g., "90% Happy") is insufficient. Stakeholders need to understand why a model reached a specific conclusion. Human experts utilize the Facial Action Coding System (FACS), which decomposes expressions into discrete Action Units (AUs)—such as a lip corner puller (AU12) or a brow lowerer (AU4).

Standard deep learning models, however, are prone to picking up "noise"—background elements, hair, or accessories—to make predictions. This lack of transparency hinders trust and deployment in sensitive applications. The goal of this work is to bridge the gap between expert anatomical knowledge and the representational power of deep neural networks.

Methodology: The Guided Interpretable Framework

The authors propose a "Guided Interpretable" framework that can be integrated into various architectures, including Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The process involves three primary components:

  1. AU Heatmap Generation: Using a predefined "codebook" that maps basic emotions to specific AUs, the system identifies relevant facial regions. Combined with automated facial landmark detection, it generates a 2D spatial heatmap (the "True AU map") for every training image.
  2. Layer-wise Attention Extraction: As the model processes an image, it generates internal attention maps at different layers, representing the features it finds most salient.
  3. Alignment via Composite Loss: The core innovation is an AU map alignment loss based on cosine similarity. During training, the model is penalized if its internal attention maps do not align with the expert-derived AU heatmaps.
Overview of the FER classifier architecture and the impact of AU-guided training on visual interpretability.

Importantly, this method requires no additional manual labeling. It leverages existing emotion class labels and off-the-shelf landmark detectors to create the guidance signal, making it highly scalable.

Key Results and Visual Interpretability

The framework was rigorously tested on the RAF-DB and AffectNet datasets. The results challenge the common assumption of an "accuracy-interpretability trade-off."

  • Superior Localization: The alignment between model attention and actual AUs improved significantly. In tests using the APViT model on RAF-DB, the attention-localization score rose from 0.38 to 0.85.
  • Maintained Accuracy: Guided models achieved competitive accuracy (~90% on RAF-DB), matching or slightly outperforming unguided baselines. This suggests that focusing on the "right" features can be as effective as unconstrained feature learning.
  • Qualitative Clarity: Visualizations demonstrate that guided models focus precisely on the eyes, nose, and mouth regions relevant to the specific emotion, whereas unguided models often produce scattered or irrelevant heatmaps.
Visualization of AU heatmaps for various facial expressions, showing the specific muscle groups targeted by the guidance system.

Discussion and Future Directions

While the framework represents a significant step toward transparent FER, some limitations remain. The system's performance is currently tied to the accuracy of the initial facial landmark detector; errors in landmark placement can lead to inaccurate AU guidance. Furthermore, the current implementation uses a "generic" AU map for each emotion category. Future work could involve using pre-trained AU detectors to identify active AUs on a per-face basis, accounting for individual variations in expression intensity.

By shifting from post-hoc explanations to intrinsic interpretability, this research provides a blueprint for building AI systems that are not only accurate but also biologically grounded and trustworthy.

Summary updated

BibTeX
@inproceedings{belharbi2024guided,
  author = {Soufiane Belharbi and Marco Pedersoli and Alessandro Lameiras Koerich and Simon Bacon and Eric Granger},
  title = {{Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues}},
  booktitle = {FG},
  year = {2024},
  doi = {10.1109/fg59268.2024.10582016},
}
Figure