Explicit Facial Expression Transfer via Fine-Grained Semantic Representations

Zhiwen Shao, Hengliang Zhu, Junshu Tang, Xuequan Lu, Lizhuang Ma

Abstract

Facial expression transfer aims to map the expression of a source image onto a target identity while preserving the target's pose and lighting. Traditional methods often rely on intermediate landmarks or global representations, which can lead to artifacts or loss of fine-grained detail. This paper introduces Explicit Facial Expression Transfer (EET), a framework that directly disentangles and swaps fine-grained Action Unit (AU) features. By employing multi-class adversarial training and a novel swap consistency loss, EET achieves high-fidelity transfer of subtle expressions, including gaze direction, while strictly maintaining identity integrity.

Introduction

The challenge of facial expression transfer lies in the inherent "tangling" of expression and identity features. Most existing models treat expression as a global attribute, failing to capture localized, subtle movements like a slight squint or lip corner lift. Furthermore, many methods depend on pre-processed landmarks or AU predictions; errors in these "middleman" steps often propagate to the final synthesis, causing "ghosting" or unnatural distortions.

The EET framework addresses these gaps by proposing a direct disentanglement-swap-generation process that operates on fine-grained semantic representations without relying on explicit landmark guidance.

Methodology

The core of the EET architecture is designed to isolate expression-specific information from identity-related features through a structured pipeline.

Fine-Grained Disentanglement

The model utilizes a feature encoder to bifurcate an image into two distinct representations:

  1. AU-related features: These capture specific muscle movements. Unlike global vector approaches, EET uses independent convolutional branches for each Action Unit, allowing the model to focus on localized facial regions.
  2. AU-free features: These encapsulate identity, pose, and texture information.
Overview of the EET training and disentanglement-swap-generation process.

Multi-Class Adversarial Training

To ensure the "AU-free" features are truly devoid of expression data, a two-stage adversarial training strategy is employed. An AU discriminator is trained to classify AU intensities. Simultaneously, the AU-free encoder is optimized to produce features that confuse the discriminator, specifically forcing it toward a uniform probability distribution across all intensity levels.

Swap Consistency Loss ($L_{sc}$)

In the absence of paired ground-truth data (e.g., the same person with different expressions), the authors introduce a swap consistency loss. This loss ensures that a self-reconstructed image and a synthesized image (with swapped expressions) remain consistent in their underlying facial structure and texture, preventing identity leakage from the source.

Experiments and Results

The framework was evaluated on several benchmark datasets, including DISFA, RaFD, MMI, and the Chicago Face Database (CFD).

Quantitative Performance

EET demonstrated superior performance in terms of Intra-class Correlation (ICC) and Mean Square Error (MSE) when measuring the accuracy of transferred AU intensities. In face verification tests using VGG-Face and LightCNN, EET maintained identity scores comparable to real images, significantly outperforming baseline models that often "leaked" source identity into the target.

Qualitative Comparison

As shown in the comparative results, EET excels at capturing fine-grained details that other models often blur or ignore. This includes the precise transfer of gaze direction and subtle micro-expressions, maintaining high visual fidelity across various subjects.

Qualitative comparison of EET against baseline models and its variants across different identities and expressions.

Conclusion

The EET framework represents a significant step forward in explicit facial expression transfer. By moving away from implicit guidance and focusing on fine-grained, disentangled AU branches, the model achieves state-of-the-art results in both expression accuracy and identity preservation. Future work will focus on improving cross-dataset generalization to handle the diverse lighting and background conditions found in "in-the-wild" imagery.

Summary updated

BibTeX
@article{shao2021explicit,
  author = {Zhiwen Shao and Hengliang Zhu and Junshu Tang and Xuequan Lu and Lizhuang Ma},
  title = {{Explicit Facial Expression Transfer via Fine-Grained Semantic Representations}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=sDf8-NAr1f},
}
Figure