TL;DR
This paper introduces Animating Through Warping (ATW), a novel framework that enables high-resolution (up to 4K) facial animation from still images, overcoming the resolution limitations of previous methods. ATW achieves this by efficiently handling high-frequency details through a residual warping mechanism, making high-quality facial animation accessible for applications requiring larger image sizes.
What Problem Does This Paper Address?
Animating still images, particularly faces, to create realistic expressions and movements has seen significant advancements with deep learning. However, a major bottleneck has been the inability of existing methods to handle high-resolution (HD) images (typically limited to 512x512 or smaller). This limitation stems from memory constraints during training and inference, the difficulty of acquiring large HD training datasets, and the computational cost associated with processing high-resolution data. Consequently, the potential applications of these methods in fields like movie production and interactive systems are significantly curtailed. This paper aims to bridge this gap by proposing an efficient framework capable of animating HD facial images without requiring HD training data or excessive computational resources.
Methods
The core of the ATW framework is a two-stage approach:
- Generator: This module takes a low-resolution input image (128x128) and a driving signal (e.g., expression category, action units) to predict:
- A low-resolution animated image.
-
A motion field that warps the input face to the desired expression. The generator itself is composed of two sub-networks: a Motion Network to predict the motion field and a Refinement Network to enhance the visual quality of the warped result.
-
Residual Warping (ResWarp) Module: This novel post-processing module is key to achieving high-resolution outputs. It works by:
- Decomposing the original HD input image into a low-resolution component and high-frequency residuals.
- Warping these high-frequency residuals using the motion field predicted by the generator.
- Adding the warped residuals to the up-sampled low-resolution animated output from the generator to produce the final high-resolution, sharp image.
The paper also introduces a multiscale ResWarp variation, which applies the warping process at multiple resolutions, leading to more realistic textures. The training methodology employs a combination of five loss terms: adversarial loss, driving loss, warp loss, refine loss, and cycle loss, to ensure realism, adherence to the driving signal, temporal consistency, and identity preservation.
Datasets
The ATW framework was trained and evaluated on the following datasets:
- CelebA: Used for both attribute-driven (smiling/non-smiling) and example-based (action units) facial animation.
- EmotionNet: Utilized for attribute-driven facial animation with 26 expression categories.
- RaFD: Used for attribute-driven facial animation with 8 expression categories.
The models were trained on 128x128 images and tested on images of various resolutions, including up to 4K.
Key Insights and Ideas
- High-Frequency Residuals are Key: The paper's central idea is that high-quality, high-resolution results can be achieved by generating a low-resolution animation and then intelligently re-applying high-frequency details. This decouples the complexity of generating facial motion from the challenge of rendering fine textures at high resolutions.
- Efficient HD Animation: ATW demonstrates that animating HD images is feasible without requiring HD training data or excessive computational resources, by training on low-resolution images and performing inference on any size input.
- Multiscale ResWarp for Better Textures: The multiscale approach to warping residuals significantly improves the realism of textures, particularly in challenging areas like the eyes, compared to a single-scale or vanilla ResWarp.
- De-coupling Motion and Appearance: The "refine loss" encourages the motion and appearance to be fully decoupled, which helps in achieving better temporal coherency and reducing color flickering during animation.
Key Results
- 4K Animation Achieved: The ATW framework successfully animates facial images up to 4K resolution, a feat not achieved by prior neural models.
- State-of-the-Art Performance: ATW outperforms existing methods like Ganimation, x2face, and few-shot vid2vid in terms of visual quality, sharpness, and identity preservation, as demonstrated by qualitative comparisons and quantitative metrics like FID and CSS.
- Efficiency: The method is computationally efficient, allowing for inference on high-resolution images with limited resources. For example, it can animate 1K images with comparable time and memory to competitors.
- Temporal Coherency: The framework generally guarantees temporal coherency in generated animations, supported by low Flow Warping Error (FWE) scores.
- Ablation Studies: Loss function ablation studies highlight the importance of each component, with the refine loss proving crucial for temporal coherency and image quality.
Limitations and Future Work
- Synthesis of Novel Contents: The method struggles with synthesizing novel contents, such as teeth when a mouth opens, leading to blurriness. This is because the corresponding residuals cannot be effectively warped from the source image.
- Motion Field Artifacts: Occasionally, the model predicts invalid motion fields, resulting in motion artifacts.
- Future Work: The authors suggest extending the framework for reenactment of head poses and arbitrary objects or scenes. They also plan to explore further optimization of hyper-parameters for specific requirements.
Summary updated