Learning to Deblur and Rotate Motion-Blurred Faces

Givi Meishvili, Attila Szabó, Simon Jenni, Paolo Favaro

TL;DR

This paper introduces a novel method for generating sharp, multi-view videos of faces from a single motion-blurred image. By leveraging a generative model of faces and a new multi-view dataset, the approach can simultaneously deblur and rotate faces, opening up new possibilities for realistic video manipulation and telepresence.

What Problem Does This Paper Address?

Capturing high-quality video of faces, especially in low-light conditions or with fast motion, often results in motion blur. This blur degrades the visual quality and makes it difficult to accurately perceive facial expressions or perform tasks like facial recognition. Furthermore, in applications like teleconferencing, it's desirable to dynamically change the viewpoint of the person on screen to improve engagement. However, existing methods struggle to both remove motion blur and synthesize novel views from a single, degraded input. This paper tackles the challenging problem of reconstructing a sharp, temporally coherent video sequence from a single motion-blurred face image and then rendering it from arbitrary viewpoints.

Methods

The core of the proposed method lies in a two-stage neural network architecture. First, a generative model (StyleGAN2) is trained to produce high-quality sharp face images. Then, an encoder network ($E_s$) is trained to invert this generator, mapping sharp images to the latent space of the generator. This latent space is crucial because it allows for easier manipulation of facial attributes, including viewpoint.

To handle motion blur, a blurry image encoder ($E_v$) is trained. This encoder takes a blurry image and predicts a sequence of latent codes corresponding to multiple sharp frames. This is achieved by using the latent codes from the sharp image encoder ($E_s$) as targets, effectively teaching $E_v$ to "see through" the blur.

For novel view synthesis, the system needs to understand the 3D pose of the face. This is accomplished by training a 3D viewpoint encoder ($E_{3D}$) that regresses the parameters of a 3D Morphable Model (3DMM). These 3DMM coefficients capture the identity, expression, and pose of the face.

Finally, a fusion network ($F$) combines the latent codes from $E_v$ with the estimated 3D pose information from $E_{3D}$. This network learns to manipulate the latent codes to represent a desired change in viewpoint, effectively rotating the face in the latent space. The modified latent codes are then fed back into the generator $G$ to produce the sharp video frames from the novel view.

Datasets

The authors introduce a new dataset called the Bern Multi-View Face Dataset (BMFD). This dataset consists of high-speed (112 fps) synchronized videos of 52 individuals captured from 8 fixed viewpoints, performing various facial expressions. The high frame rate is essential for simulating realistic motion blur through temporal averaging.

In addition to BMFD, the method is trained and evaluated on publicly available datasets:

  • FFHQ: Used for its large variety of faces to improve generalization.
  • 300VW: Also used for training data synthesis.
  • VIDTIMIT: Used for evaluation of the deblurring and novel view synthesis capabilities.

Key Insights and Ideas

  • Latent Space Manipulation for Viewpoint Control: The paper highlights the power of manipulating latent codes in generative models like StyleGAN2 for controlling facial attributes, including viewpoint, which is more effective than direct image manipulation.
  • Leveraging a Pre-trained Generator as a Teacher: Using a pre-trained sharp image generator as a "teacher" for the blurry image encoder is an effective way to learn to deblur by mapping to a well-behaved latent space.
  • 3DMM for Pose Estimation: Integrating 3D Morphable Models provides a robust way to estimate facial pose, which is critical for accurate novel view synthesis.
  • BMFD for Realistic Motion Blur and Multi-View Constraints: The newly introduced BMFD dataset is crucial for training the model to handle realistic motion blur and learn multi-view constraints, which are often missing in existing datasets.

Key Results

  • The method successfully reconstructs sharp video sequences from single motion-blurred face images.
  • It can synthesize novel views of these reconstructed faces, demonstrating effective rotation capabilities.
  • Quantitative results show competitive performance on landmark error for pose regression and identity preservation compared to prior work, even when starting from blurry inputs. For instance, the mean landmark error is reported as 3.46 pixels.
  • Qualitative results (Figures 4, 5, and supplementary figures) showcase impressive visual quality, with sharp and consistent reconstructions across different views and frames.

Limitations and Future Work

  • Ambiguity in Frame Order: The paper acknowledges the ambiguity in regressing a sequence of sharp frames from a blurry image and addresses it by allowing for both forward and backward solutions during training. However, this inherent ambiguity might still pose challenges in certain scenarios.
  • Dataset Limitations: While BMFD is a valuable contribution, the paper does not explicitly discuss limitations related to the diversity of expressions, lighting conditions, or occlusions within the dataset.
  • Future Work: The authors suggest that their dataset can be used for other tasks like 3D reconstruction, super-resolution, and temporal frame interpolation, indicating potential future research directions. The method's ability to handle complex facial expressions and dynamic scenes could also be explored further.

Summary updated

Abstract

We propose a solution to the novel task of rendering sharp videos from new viewpoints from a single motion-blurred image of a face. Our method handles the complexity of face blur by implicitly learning the geometry and motion of faces through the joint training on three large datasets: FFHQ and 300VW, which are publicly available, and a new Bern Multi-View Face Dataset (BMFD) that we built. The first two datasets provide a large variety of faces and allow our model to generalize better. BMFD instead allows us to introduce multi-view constraints, which are crucial to synthesizing sharp videos from a new camera view. It consists of high frame rate synchronized videos from multiple views of several subjects displaying a wide range of facial expressions. We use the high frame rate videos to simulate realistic motion blur through averaging. Thanks to this dataset, we train a neural network to reconstruct a 3D video representation from a single image and the corresponding face gaze. We then provide a camera viewpoint relative to the estimated gaze and the blurry image as input to an encoder-decoder network to generate a video of sharp frames with a novel camera viewpoint. We demonstrate our approach on test subjects of our multi-view dataset and VIDTIMIT.
BibTeX
@misc{meishvili2021learning,
  author = {Givi Meishvili and Attila Szabó and Simon Jenni and Paolo Favaro},
  title = {{Learning to Deblur and Rotate Motion-Blurred Faces}},
  year = {2021},
  eprint = {2112.07599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.07599},
}
Figure