Context-Aware Feature and Label Fusion for Facial Action Unit Intensity Estimation With Partially Labeled Data

Yong Zhang, Haiyong Jiang, Baoyuan Wu, Yanbo Fan, Qiang Ji

Introduction

Accurately estimating the intensity of facial expressions is a fundamental task in affective computing. Facial Action Units (AUs) represent the movements of individual or groups of facial muscles. While detecting the presence of an AU is relatively straightforward, estimating its intensity on a multi-point scale is significantly more challenging due to the subtle visual differences between levels. Traditionally, this task has required massive amounts of fully annotated video data, where every frame is labeled by experts—a process that is both time-consuming and expensive.

The Annotation Bottleneck

A major hurdle in facial expression analysis is the reliance on "fully supervised" models. These models typically require every frame in a dataset to be annotated. Furthermore, many existing "patch-based" approaches treat all facial regions with equal importance. However, facial expressions are localized; for instance, a "lip corner puller" (AU 12) is primarily visible around the mouth, making eye-region patches irrelevant for its estimation.

Proposed Methodology: The CFLF Framework

To address these challenges, researchers have developed the Context-aware Feature and Label Fusion (CFLF) framework. This weakly supervised model achieves state-of-the-art performance using only 1–2% of the labels required by traditional methods by focusing on "peak" (maximum intensity) and "valley" (minimum intensity) frames.

Overview of the CFLF framework showing the training and inference pipelines.

1. Context-Aware Feature Fusion

The core innovation lies in making the model "aware" of the specific task at hand. Instead of a static feature extraction process, the model utilizes a learnable task context. This context vector represents the specific AU being estimated and guides an attention mechanism to weight facial patches differently.

Detailed architecture of the Context-Aware Feature Fusion module, illustrating how task context influences patch attention.

As shown in the architecture, local patches (processed via CNN-L1 to CNN-Ln) and global facial features (CNN-G) are fused. The task context allows the model to "attend" to the most relevant local regions for a given AU, mimicking the focus of human annotators.

2. Context-Aware Label Fusion

To handle the temporal gaps between labeled peak and valley frames, the framework employs a Context-Aware Label Fusion module. This module uses Long Short-Term Memory (LSTM) networks to model the temporal dynamics of facial muscle movements. By incorporating the task context, the LSTM can adapt its temporal modeling to the specific "evolution rules" of different AUs, effectively distributing sequence-level information across unlabeled frames.

Experimental Results

The CFLF model was evaluated on two benchmark datasets: FERA 2015 and DISFA. Despite using only the peak and valley annotations—representing approximately 2% and 1% of the total frames respectively—the model demonstrated remarkable efficiency and accuracy.

  • Performance: On FERA 2015, the model achieved an average Intra-class Correlation (ICC) of 0.661, outperforming other weakly supervised baselines such as KBSS (0.645) and BORMIR (0.620).
  • Efficiency: The framework surpassed several fully supervised state-of-the-art models (e.g., ResNet18) while requiring roughly 98% fewer labels.
  • Ablation Insights: Studies confirmed that the "task context" is critical; removing it led to a significant drop in estimation accuracy, proving that spatial and temporal attention must be task-dependent.

Conclusion and Future Directions

The CFLF framework demonstrates that smart, context-aware fusion can overcome the limitations of sparse data in facial expression analysis. By intelligently weighting spatial patches and temporal moments based on the specific Action Unit, the model achieves high precision with minimal supervision. Future work may explore the model's robustness against significant head pose variations and the potential for joint estimation of correlated AUs to further enhance intensity predictions.

Summary updated

Abstract

Facial action unit (AU) intensity estimation is a fundamental task for facial behaviour analysis. Most previous methods use a whole face image as input for intensity prediction. Considering that AUs are defined according to their corresponding local appearance, a few patch-based methods utilize image features of local patches. However, fusion of local features is always performed via straightforward feature concatenation or summation. Besides, these methods require fully annotated databases for model learning, which is expensive to acquire. In this paper, we propose a novel weakly supervised patch-based deep model on basis of two types of attention mechanisms for joint intensity estimation of multiple AUs. The model consists of a feature fusion module and a label fusion module. And we augment attention mechanisms of these two modules with a learnable task-related context, as one patch may play different roles in analyzing different AUs and each AU has its own temporal evolution rule. The context-aware feature fusion module is used to capture spatial relationships among local patches while the context-aware label fusion module is used to capture the temporal dynamics of AUs. The latter enables the model to be trained on a partially annotated database. Experimental evaluations on two benchmark expression databases demonstrate the superior performance of the proposed method.
BibTeX
@inproceedings{zhang2019context,
  author = {Yong Zhang and Haiyong Jiang and Baoyuan Wu and Yanbo Fan and Qiang Ji},
  title = {{Context-Aware Feature and Label Fusion for Facial Action Unit Intensity Estimation With Partially Labeled Data}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00082},
}
Figure