AU-Aware Dynamic 3D Face Reconstruction From Videos With Transformer

Chenyi Kuang, Jeffrey O. Kephart, Qiang JiWACV 2024

TL;DR

This paper introduces an "AU-Aware Dynamic 3D Face Reconstruction" framework that leverages transformers to capture the subtle temporal dynamics and spatial correlations of facial Action Units (AUs) from monocular videos. This approach leads to more accurate and stable 3D face reconstructions, especially for complex expressions, and simultaneously improves AU recognition.

What Problem Does This Paper Address?

Reconstructing accurate and stable 3D faces from videos, particularly capturing nuanced facial expressions, remains a significant challenge. Existing methods often struggle with temporal consistency, leading to unnatural or jittery results, especially during rapid expression changes. Furthermore, many approaches don't fully leverage the underlying muscle activation system (represented by Action Units) that drives facial movements, nor do they effectively model the spatial relationships between different AUs on the face. This limits their ability to capture the full expressiveness and dynamism of human faces.

Methods

The proposed method builds upon a coarse-to-fine strategy. First, a pre-trained static 3D face reconstruction model (DECA) is used to generate initial, frame-by-frame coarse 3D face meshes. These coarse meshes, along with video features extracted by a backbone model (Video Swin Transformer), are then fed into a novel Transformer-based framework.

The core innovation lies in two Transformer modules:

  • Temporal Module: This module uses an autoregressive Transformer (similar to GPT) to model the temporal dependencies of facial motion dynamics. It takes AU label embeddings and temporal features as input to predict temporal embeddings for each frame, capturing the sequence of expression changes.
  • Spatial Module: This module models the spatial correlations among AUs from the 3D geometry. It takes geometry-based AU tokens (derived from the coarse 3D meshes) and uses a Transformer encoder to learn how different AUs interact spatially on the face, producing spatial embeddings.

These temporal and spatial embeddings are then combined and fed into a prediction head that simultaneously generates refined 3D facial meshes and predicts AU occurrence probabilities.

Datasets

The paper uses the following datasets:

  • BP4D [59]: A spontaneous database with 328 sequences from 41 subjects, featuring binary AU labels for training and evaluation.
  • DISFA [40]: Contains videos from 27 subjects with AU intensity labels (0-5), used for AU recognition evaluation.
  • Multi-face [54]: A large-scale, multi-view dataset with synchronized 2D and 3D videos, used for evaluating dynamic 3D reconstruction accuracy. A subset was used for experiments.

Key Insights and Ideas

  • Leveraging AU Dynamics: The paper demonstrates that explicitly modeling the temporal dynamics and spatial correlations of Action Units is crucial for achieving high-quality, dynamic 3D face reconstruction.
  • Transformer for Spatio-Temporal Facial Modeling: The application of Transformer architectures, particularly in an autoregressive manner for temporal modeling and with self-attention for spatial correlations, proves highly effective for capturing complex facial expressions.
  • Coarse-to-Fine Refinement: Starting with a static reconstruction and then refining it with dynamic, AU-aware information provides a robust pipeline for handling video data.
  • Synergy of Appearance and Geometry: Combining video appearance features with 3D mesh geometry features is essential for comprehensive facial understanding and reconstruction.

Key Results

  • Reconstruction Accuracy: The proposed "Temporal + Spatial" model achieved uniformly lower reconstruction error (vertex-to-surface MSE and SD) on the Multiface dataset compared to DECA [17] and EMOCA [9], reducing 3D error by 7.45% compared to EMOCA.
  • AU Recognition: On BP4D, the model achieved an average F1 score improvement of 1.9% over Biomechanics-AU [8] and demonstrated superiority over appearance-based methods. It achieved SOTA accuracy on specific AUs (AU4, AU10, AU17, AU24). On DISFA, it showed significant improvements for certain AUs (e.g., AU6 +13.9%, AU9 +20.2%, AU12 +14.1%).
  • Qualitative Results: Visual comparisons (Figure 3) show that the proposed method better captures subtle motions related to AUs, such as eye-brow frowning, nose wrinkling, and mouth dimpling, leading to more natural and stable reconstructions.

Limitations and Future Work

  • AU Occurrence Embeddings: The temporal module's reliance on binary AU occurrence embeddings limits its ability to capture more nuanced dynamics beyond simple state changes (0 to 1 or 1 to 0).
  • Frame Rate Dependency: The performance of the dynamic model can be impacted by different frame rates (fps) of the input videos.
  • Future Work: The authors suggest exploring more sophisticated ways to represent AU dynamics beyond binary labels and investigating methods to make the model less sensitive to input frame rates.

Summary updated

BibTeX
@inproceedings{kuang2024au,
  author = {Chenyi Kuang and Jeffrey O. Kephart and Qiang Ji},
  title = {{AU-Aware Dynamic 3D Face Reconstruction From Videos With Transformer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure