Lifting Scheme-Based Implicit Disentanglement of Emotion-Related Facial Dynamics in the Wild

Xingjian Wang, Li ChaiAAAI 2025 · DOI

TL;DR

This paper introduces the Implicit Facial Dynamics Disentanglement (IFDD) framework to better recognize dynamic facial expressions in real-world videos. IFDD uses a novel wavelet lifting-based approach to implicitly separate emotion-related facial movements from irrelevant background and identity information, leading to improved accuracy.

What Problem Does This Paper Address?

Recognizing dynamic facial expressions in real-world videos (in-the-wild DFER) is significantly harder than in controlled lab settings. This is because emotion-related expressions are often diluted by many neutral or irrelevant frames, and can be masked by background noise or individual facial characteristics. Existing methods often struggle with this by directly using coupled spatiotemporal features, which can inadvertently incorporate these irrelevant cues, leading to biased and less accurate emotion recognition. While some methods try to focus on dynamic information, they often rely on explicit guidance (like optical flow or facial landmarks) that can be sensitive to noise and head movements.

Methods

The core of IFDD is a two-stage framework that implicitly disentangles emotion-related dynamics from global context:

  • Inter-frame Static-dynamic Splitting Module (ISSM): This module first processes the input video features to generate "content-aware splitting indexes." Unlike traditional fixed splitting methods, ISSM adaptively learns these indexes based on temporal correlations within the video frames. This allows it to roughly split the features into two groups: one representing more static, globally similar information, and another capturing more unique, potentially emotion-related dynamic features.
  • Lifting-based Aggregation-Disentanglement Module (LADM): This module refines the initial split. It first aggregates the two feature groups to create a more robust global context representation. Then, using a predictor, it subtracts this global context from the dynamic features, effectively purifying them to isolate the emotion-related dynamics. This process is inspired by wavelet lifting schemes, which are adapted to be fully learnable within the neural network.

The framework is designed to be backbone-agnostic, meaning it can be integrated with various convolutional neural networks (CNNs) or Vision Transformers (ViTs). A key aspect is the "implicit" nature of the disentanglement, meaning it doesn't rely on explicit external guidance like optical flow.

Datasets

The paper evaluates IFDD on three widely used in-the-wild DFER datasets:

  • DFEW: Contains 16,372 videos with 7 emotion categories.
  • FERV39k: Contains 38,935 videos with 7 emotion categories.
  • MAFW: Contains 10,045 videos with 11 emotion categories.

These datasets are challenging because they feature limited emotion-relevant frames, temporal dilution, occlusions, and pose variations, making them representative of real-world scenarios.

Key Insights and Ideas

  • Implicit Disentanglement is Key: The paper argues that explicitly guiding the model to find "dynamics" can be brittle. Instead, implicitly separating emotion-related dynamics from global context is more robust for in-the-wild scenarios.
  • Adaptive Splitting Outperforms Fixed: The ISSM's ability to learn adaptive splitting indexes based on temporal correlation is crucial for effectively separating static and dynamic features, leading to better performance than fixed even-odd splitting.
  • Two-Stage Refinement Works: The combination of ISSM for initial rough splitting and LADM for aggregation and refinement provides a powerful mechanism for isolating the desired dynamic features.
  • Wavelet Lifting as a Learnable Framework: Adapting the wavelet lifting scheme into a fully learnable module within a deep learning architecture offers a novel way to process and disentangle features.

Key Results

IFDD significantly outperforms state-of-the-art supervised DFER methods across the evaluated datasets.

  • On the DFEW dataset, IFDD-3DViT achieved a Weighted Average Recall (WAR) of 73.82%, which is 6.60% higher than the previous best supervised method (M3DFEL).
  • On FERV39k, IFDD-3DViT outperformed the second-best method by 7.17% in WAR.
  • On MAFW, IFDD-3DViT showed substantial gains, exceeding the second-best method by 11.91% in WAR.
  • The paper also highlights the efficiency of IFDD, noting that even with performance gains, the increase in computational cost (FLOPs) is relatively small.

Limitations and Future Work

  • Limitations: The paper acknowledges that while IFDD is implicit, it still relies on a backbone network for initial feature extraction. The performance is thus dependent on the chosen backbone.
  • Future Work: The authors suggest exploring the IFDD framework for other dynamic-sensitive tasks, such as dynamic micro-expression recognition, optical flow estimation, and video compression. They also mention the potential for further research into its compatibility with different backbones and its application in self-supervised learning settings.

Summary updated

BibTeX
@inproceedings{wang2025lifting,
  author = {Xingjian Wang and Li Chai},
  title = {{Lifting Scheme-Based Implicit Disentanglement of Emotion-Related Facial Dynamics in the Wild}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32859},
}
Figure