TL;DR
This paper introduces a novel method for generating sharp, multi-view videos of faces from a single motion-blurred image. By leveraging a generative model of faces and a new multi-view dataset, the approach can simultaneously deblur and rotate faces, opening up new possibilities for realistic video manipulation and telepresence.
What Problem Does This Paper Address?
Capturing high-quality video of faces, especially in low-light conditions or with fast motion, often results in motion blur. This blur degrades the visual quality and makes it difficult to accurately perceive facial expressions or perform tasks like facial recognition. Furthermore, in applications like teleconferencing, it's desirable to dynamically change the viewpoint of the person on screen to improve engagement. However, existing methods struggle to both remove motion blur and synthesize novel views from a single, degraded input. This paper tackles the challenging problem of reconstructing a sharp, temporally coherent video sequence from a single motion-blurred face image and then rendering it from arbitrary viewpoints.
Methods
The core of the proposed method lies in a two-stage neural network architecture. First, a generative model (StyleGAN2) is trained to produce high-quality sharp face images. Then, an encoder network ($E_s$) is trained to invert this generator, mapping sharp images to the latent space of the generator. This latent space is crucial because it allows for easier manipulation of facial attributes, including viewpoint.
To handle motion blur, a blurry image encoder ($E_v$) is trained. This encoder takes a blurry image and predicts a sequence of latent codes corresponding to multiple sharp frames. This is achieved by using the latent codes from the sharp image encoder ($E_s$) as targets, effectively teaching $E_v$ to "see through" the blur.
For novel view synthesis, the system needs to understand the 3D pose of the face. This is accomplished by training a 3D viewpoint encoder ($E_{3D}$) that regresses the parameters of a 3D Morphable Model (3DMM). These 3DMM coefficients capture the identity, expression, and pose of the face.
Finally, a fusion network ($F$) combines the latent codes from $E_v$ with the estimated 3D pose information from $E_{3D}$. This network learns to manipulate the latent codes to represent a desired change in viewpoint, effectively rotating the face in the latent space. The modified latent codes are then fed back into the generator $G$ to produce the sharp video frames from the novel view.
Datasets
The authors introduce a new dataset called the Bern Multi-View Face Dataset (BMFD). This dataset consists of high-speed (112 fps) synchronized videos of 52 individuals captured from 8 fixed viewpoints, performing various facial expressions. The high frame rate is essential for simulating realistic motion blur through temporal averaging.
In addition to BMFD, the method is trained and evaluated on publicly available datasets:
- FFHQ: Used for its large variety of faces to improve generalization.
- 300VW: Also used for training data synthesis.
- VIDTIMIT: Used for evaluation of the deblurring and novel view synthesis capabilities.
Key Insights and Ideas
- Latent Space Manipulation for Viewpoint Control: The paper highlights the power of manipulating latent codes in generative models like StyleGAN2 for controlling facial attributes, including viewpoint, which is more effective than direct image manipulation.
- Leveraging a Pre-trained Generator as a Teacher: Using a pre-trained sharp image generator as a "teacher" for the blurry image encoder is an effective way to learn to deblur by mapping to a well-behaved latent space.
- 3DMM for Pose Estimation: Integrating 3D Morphable Models provides a robust way to estimate facial pose, which is critical for accurate novel view synthesis.
- BMFD for Realistic Motion Blur and Multi-View Constraints: The newly introduced BMFD dataset is crucial for training the model to handle realistic motion blur and learn multi-view constraints, which are often missing in existing datasets.
Key Results
- The method successfully reconstructs sharp video sequences from single motion-blurred face images.
- It can synthesize novel views of these reconstructed faces, demonstrating effective rotation capabilities.
- Quantitative results show competitive performance on landmark error for pose regression and identity preservation compared to prior work, even when starting from blurry inputs. For instance, the mean landmark error is reported as 3.46 pixels.
- Qualitative results (Figures 4, 5, and supplementary figures) showcase impressive visual quality, with sharp and consistent reconstructions across different views and frames.
Limitations and Future Work
- Ambiguity in Frame Order: The paper acknowledges the ambiguity in regressing a sequence of sharp frames from a blurry image and addresses it by allowing for both forward and backward solutions during training. However, this inherent ambiguity might still pose challenges in certain scenarios.
- Dataset Limitations: While BMFD is a valuable contribution, the paper does not explicitly discuss limitations related to the diversity of expressions, lighting conditions, or occlusions within the dataset.
- Future Work: The authors suggest that their dataset can be used for other tasks like 3D reconstruction, super-resolution, and temporal frame interpolation, indicating potential future research directions. The method's ability to handle complex facial expressions and dynamic scenes could also be explored further.
Summary updated