Explicit Facial Expression Transfer via Fine-Grained Representations

Zhiwen Shao, Hengliang Zhu, Junshu Tang, Xuequan Lu, Lizhuang Ma

This academic paper, published in IEEE Transactions on Image Processing, presents a novel framework for Explicit Facial Expression Transfer (EET). The authors move away from traditional methods that rely on intermediate "guidance" (like predicted landmarks) and instead propose a system that directly disentangles and swaps fine-grained facial representations.

TL;DR

The authors introduce EET, a deep learning framework that transfers subtle facial expressions between unpaired images by disentangling "Action Unit" (AU) features from identity features. By using a novel multi-class adversarial training and a swap consistency loss, the model achieves high-fidelity expression transfer—including fine details like gaze direction—while strictly preserving the target's identity and pose.

What Problem Does This Paper Address?

Facial expression transfer is the task of taking the "expression" from a source image and applying it to a target image while keeping the target's identity, pose, and lighting intact.

The authors identify two major gaps in existing research:

  1. Tangled Representations: Expressions are often "tangled" with identity. Most models treat expression transfer as a global manipulation (e.g., "make this person happy"), which fails to capture fine-grained details like a slight squint or a specific lip movement.
  2. The "Middleman" Problem: Many current methods rely on offline pre-processing to predict landmarks or Action Units (AUs) to guide the synthesis. If these initial predictions are even slightly off, the final generated image suffers from artifacts or "ghosting."

Methods

The core of the EET framework is a disentanglement-swap-generation process.

1. Fine-Grained Disentanglement

The model uses a feature encoder to split an image into two distinct representations:

  • AU-related features: Capturing the specific muscle movements (Action Units). To ensure these are truly fine-grained, the authors use independent branches for each AU rather than one global vector.
  • AU-free features: Capturing identity, pose, and texture.

2. Multi-Class Adversarial Training

To ensure the "AU-free" features truly contain zero expression information, the authors propose a two-stage adversarial training. They train an AU discriminator to classify AU intensities. Simultaneously, they train the AU-free encoder to produce features that make the discriminator "confused"—specifically, forcing it to output a uniform probability distribution across all possible AU intensity levels.

3. Swap Consistency Loss ($L_{sc}$)

Since there is no "ground truth" for what Person A looks like with Person B’s expression, the authors introduce a swap consistency loss. This ensures that the synthesized image (Person A with Person B's expression) and a self-reconstructed image (Person A with Person A's expression) remain indistinguishable in terms of underlying facial structure and texture.

Datasets

The model was trained and evaluated on four prominent facial datasets:

  • DISFA: Contains 27 subjects with frame-by-frame annotations for 12 Action Units on a 6-point intensity scale.
  • RaFD: Includes 67 subjects with 8 global expressions and 3 gaze directions.
  • MMI: A dataset of over 2,000 videos/images, some featuring occlusions like eyeglasses.
  • CFD (Chicago Face Database): Used to test the model's robustness across different races and ages.

Note: The authors used the OpenFace library to provide AU annotations for datasets that lacked them.

Key Insights and Ideas

  • Explicit is Better than Implicit: By mapping images directly to swapped expressions without an intermediate landmark-prediction step, the model avoids the "compounding error" problem where a bad landmark leads to a bad face.
  • Local Branches for Local Actions: The decision to use independent convolutional branches for each AU is crucial. It allows the model to focus on specific regions (like the eyes for gaze or the mouth for smiles) rather than trying to compress the entire face's motion into a single bottleneck.
  • Adversarial Confusion: The use of multi-class adversarial training to "strip" information from a feature vector is a clever way to achieve clean disentanglement. It’s more robust than simple binary (real/fake) adversarial losses.

Key Results

The EET framework was compared against state-of-the-art models like StarGAN, GANimation, and HomoInterpGAN.

  • Quantitative: EET achieved the highest Intra-class Correlation (ICC) and lowest Mean Square Error (MSE) when measuring the accuracy of transferred AU intensities.
  • Qualitative: EET significantly outperformed others in transferring "fine-grained" details. For example, it successfully transferred gaze direction (looking left/right) and subtle lip corner lifts that other models ignored or blurred.
  • Identity Preservation: In face verification tests (using VGG-Face and LightCNN), EET maintained identity scores comparable to real images, whereas competitors often "leaked" the source person's identity into the target image.

Limitations and Future Work

The authors acknowledge a primary limitation: Cross-dataset generalization. When a model trained on one dataset (like RaFD) is tested on a completely different one (like DISFA), artifacts can appear on the skin texture. This is due to the limited diversity in the training data's lighting and background conditions.

Future Work: The authors suggest exploring ways to solve this "cross-dataset testing issue" to make the model more robust for "in-the-wild" images where lighting and textures vary wildly.

Summary updated

Abstract

Facial expression transfer between two unpaired images is a challenging problem, as fine-grained expression is typically tangled with other facial attributes. Most existing methods treat expression transfer as an application of expression manipulation, and use predicted global expression, landmarks or action units (AUs) as a guidance. However, the prediction may be inaccurate, which limits the performance of transferring fine-grained expression. Instead of using an intermediate estimated guidance, we propose to explicitly transfer facial expression by directly mapping two unpaired input images to two synthesized images with swapped expressions. Specifically, considering AUs semantically describe fine-grained expression details, we propose a novel multi-class adversarial training method to disentangle input images into two types of fine-grained representations: AU-related feature and AU-free feature. Then, we can synthesize new images with preserved identities and swapped expressions by combining AU-free features with swapped AU-related features. Moreover, to obtain reliable expression transfer results of the unpaired input, we introduce a swap consistency loss to make the synthesized images and self-reconstructed images indistinguishable. Extensive experiments show that our approach outperforms the state-of-the-art expression manipulation methods for transferring fine-grained expressions while preserving other attributes including identity and pose.
BibTeX
@article{shao2021explicit,
  author = {Zhiwen Shao and Hengliang Zhu and Junshu Tang and Xuequan Lu and Lizhuang Ma},
  title = {{Explicit Facial Expression Transfer via Fine-Grained Representations}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=sDf8-NAr1f},
}
Figure