Animating Through Warping: An Efficient Method for High-Quality Facial Expression Animation

Qiang Tang, Zili Yi, Qiang Tang 0002, Vishnu Sanjay Ramiya Srinivasan, Zhan Xu

TL;DR

This paper introduces Animating Through Warping (ATW), a novel framework that enables high-resolution (up to 4K) facial animation from still images, overcoming the resolution limitations of previous methods. ATW achieves this by efficiently handling high-frequency details through a residual warping mechanism, making high-quality facial animation accessible for applications requiring larger image sizes.

What Problem Does This Paper Address?

Animating still images, particularly faces, to create realistic expressions and movements has seen significant advancements with deep learning. However, a major bottleneck has been the inability of existing methods to handle high-resolution (HD) images (typically limited to 512x512 or smaller). This limitation stems from memory constraints during training and inference, the difficulty of acquiring large HD training datasets, and the computational cost associated with processing high-resolution data. Consequently, the potential applications of these methods in fields like movie production and interactive systems are significantly curtailed. This paper aims to bridge this gap by proposing an efficient framework capable of animating HD facial images without requiring HD training data or excessive computational resources.

Methods

The core of the ATW framework is a two-stage approach:

  • Generator: This module takes a low-resolution input image (128x128) and a driving signal (e.g., expression category, action units) to predict:
  • A low-resolution animated image.
  • A motion field that warps the input face to the desired expression. The generator itself is composed of two sub-networks: a Motion Network to predict the motion field and a Refinement Network to enhance the visual quality of the warped result.

  • Residual Warping (ResWarp) Module: This novel post-processing module is key to achieving high-resolution outputs. It works by:

  • Decomposing the original HD input image into a low-resolution component and high-frequency residuals.
  • Warping these high-frequency residuals using the motion field predicted by the generator.
  • Adding the warped residuals to the up-sampled low-resolution animated output from the generator to produce the final high-resolution, sharp image.

The paper also introduces a multiscale ResWarp variation, which applies the warping process at multiple resolutions, leading to more realistic textures. The training methodology employs a combination of five loss terms: adversarial loss, driving loss, warp loss, refine loss, and cycle loss, to ensure realism, adherence to the driving signal, temporal consistency, and identity preservation.

Datasets

The ATW framework was trained and evaluated on the following datasets:

  • CelebA: Used for both attribute-driven (smiling/non-smiling) and example-based (action units) facial animation.
  • EmotionNet: Utilized for attribute-driven facial animation with 26 expression categories.
  • RaFD: Used for attribute-driven facial animation with 8 expression categories.

The models were trained on 128x128 images and tested on images of various resolutions, including up to 4K.

Key Insights and Ideas

  • High-Frequency Residuals are Key: The paper's central idea is that high-quality, high-resolution results can be achieved by generating a low-resolution animation and then intelligently re-applying high-frequency details. This decouples the complexity of generating facial motion from the challenge of rendering fine textures at high resolutions.
  • Efficient HD Animation: ATW demonstrates that animating HD images is feasible without requiring HD training data or excessive computational resources, by training on low-resolution images and performing inference on any size input.
  • Multiscale ResWarp for Better Textures: The multiscale approach to warping residuals significantly improves the realism of textures, particularly in challenging areas like the eyes, compared to a single-scale or vanilla ResWarp.
  • De-coupling Motion and Appearance: The "refine loss" encourages the motion and appearance to be fully decoupled, which helps in achieving better temporal coherency and reducing color flickering during animation.

Key Results

  • 4K Animation Achieved: The ATW framework successfully animates facial images up to 4K resolution, a feat not achieved by prior neural models.
  • State-of-the-Art Performance: ATW outperforms existing methods like Ganimation, x2face, and few-shot vid2vid in terms of visual quality, sharpness, and identity preservation, as demonstrated by qualitative comparisons and quantitative metrics like FID and CSS.
  • Efficiency: The method is computationally efficient, allowing for inference on high-resolution images with limited resources. For example, it can animate 1K images with comparable time and memory to competitors.
  • Temporal Coherency: The framework generally guarantees temporal coherency in generated animations, supported by low Flow Warping Error (FWE) scores.
  • Ablation Studies: Loss function ablation studies highlight the importance of each component, with the refine loss proving crucial for temporal coherency and image quality.

Limitations and Future Work

  • Synthesis of Novel Contents: The method struggles with synthesizing novel contents, such as teeth when a mouth opens, leading to blurriness. This is because the corresponding residuals cannot be effectively warped from the source image.
  • Motion Field Artifacts: Occasionally, the model predicts invalid motion fields, resulting in motion artifacts.
  • Future Work: The authors suggest extending the framework for reenactment of head poses and arbitrary objects or scenes. They also plan to explore further optimization of hyper-parameters for specific requirements.

Summary updated

Abstract

Advances in deep neural networks have considerably improved the art of animating a still image without operating in 3D domain. Whereas, prior arts can only animate small images (typically no larger than 512x512) due to memory limitations, difficulty of training and lack of high-resolution (HD) training datasets, which significantly reduce their potential for applications in movie production and interactive systems. Motivated by the idea that HD images can be generated by adding high-frequency residuals to low-resolution results produced by a neural network, we propose a novel framework known as Animating Through Warping (ATW) to enable efficient animation of HD images. Specifically, the proposed framework consists of two modules, a novel two-stage neural-network generator and a novel post-processing module known as Animating Through Warping (ATW). It only requires the generator to be trained on small images and can do inference on an image of any size. During inference, an HD input image is decomposed into a low-resolution component(128x128) and its corresponding high-frequency residuals. The generator predicts the low-resolution result as well as the motion field that warps the input face to the desired status (e.g., expressions categories or action units). Finally, the ResWarp module warps the residuals based on the motion field and adding the warped residuals to generates the final HD results from the naively up-sampled low-resolution results. Experiments show the effectiveness and efficiency of our method in generating high-resolution animations. Our proposed framework successfully animates a 4K facial image, which has never been achieved by prior neural models. In addition, our method generally guarantee the temporal coherency of the generated animations. Source codes will be made publicly available.
BibTeX
@inproceedings{tang2020animating,
  author = {Qiang Tang and Zili Yi and Qiang Tang and Vishnu Sanjay Ramiya Srinivasan and Zhan Xu},
  title = {{Animating Through Warping: An Efficient Method for High-Quality Facial Expression Animation}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413926},
}
Figure