Deep Face Video Inpainting via UV Mapping

Wenqi Yang, Zhenfang Chen, Chaofeng Chen, Guanying Chen, Kwan-Yee K. WongIEEE TIP 2021 · DOI · OpenReview

Abstract

Video inpainting for human faces remains a significant challenge due to the non-rigid nature of facial expressions and the complex 3D transformations occurring during head rotation. Traditional 2D-based methods often struggle to maintain temporal consistency, leading to artifacts like "melting" or "ghosting." This paper introduces a novel deep learning framework that utilizes 3D facial priors to transform the inpainting task into a canonical UV texture space. By aligning facial features across frames, the model can precisely borrow missing details from neighboring frames. The proposed two-stage pipeline, consisting of a Multi-reference UV-map Completion Network (MUC-Net) and a Face Video Refinement Network (FVR-Net), achieves state-of-the-art performance on the 300VW dataset, maintaining high-quality reconstructions even under extreme poses and static occlusions.

Introduction

Video inpainting—the process of removing unwanted objects or repairing corrupted regions in a video—is relatively mature for static backgrounds. However, human faces present a unique set of difficulties. Standard methods rely on finding pixel correspondences across frames. When a subject turns their head or changes expression, the 2D appearance of facial features changes drastically, making accurate matching nearly impossible for traditional 2D algorithms. Existing models often produce "ghosting" artifacts because they fail to account for the underlying 3D structure of the head, treating the face as a flat texture.

Methodology

The researchers propose a two-stage pipeline that leverages a 3D Morphable Model (3DMM) as a structural guide to overcome the limitations of 2D-based inpainting.

Stage I: UV-Map Completion

The system first fits a 3D model to the face in every frame and "unwraps" the face into a 2D UV space. In this canonical space, facial features like the eyes, nose, and mouth are always located at the same coordinates, regardless of the person's pose or expression. A Multi-reference UV-map Completion Network (MUC-Net) then fills in the occluded regions. By using a Frame-wise Attention (FA) module, the network can easily find missing data from neighboring frames where the features are now spatially aligned.

Overview of the proposed two-stage framework, featuring 3DMM-guided UV mapping and specialized refinement networks.

Stage II: Face Video Refinement

Once the UV texture is completed, it is mapped back onto the 3D model and projected into the original 2D video frame. A second network, FVR-Net, handles the final refinement. This stage is crucial for smoothing the "seams" where the reconstructed face meets the background and ensuring that lighting and textures appear natural and temporally consistent in the original image space.

Experimental Setup

The method was evaluated using the 300VW (300 Videos in the Wild) dataset, which contains 114 videos featuring diverse poses, expressions, and lighting conditions. The authors used 75 videos for training and 20 for evaluation. To rigorously test the model's pose-invariance, they extracted every 10th frame to create sequences with significant movement. Testing included both "Shifting Masks" (moving occlusions) and "Static Masks" (fixed occlusions), the latter of which is particularly challenging for 2D methods that rely on finding clean reference pixels in other frames.

Key Results and Insights

The proposed method consistently outperformed state-of-the-art (SOTA) models such as STTN and FuseFormer across multiple metrics, including PSNR (Peak Signal-to-Noise Ratio) and VFID (Video Frechet Inception Distance).

  • Alignment as the Secret Sauce: By shifting the problem from 2D image space to 3D UV space, the authors converted a complex non-rigid motion problem into a simpler texture completion task.
  • Exploiting Symmetry: The model utilizes facial symmetry by horizontally flipping the UV map, allowing it to use a visible left eye to help reconstruct a masked right eye.
  • Qualitative Superiority: As shown in comparative evaluations, the UV-mapping approach succeeds in scenarios where a person's face is partially covered by a static block for the entire video—a case where 2D methods typically fail due to a lack of clean reference data.
Qualitative comparison of the proposed method against state-of-the-art video inpainting techniques, demonstrating superior reconstruction under occlusion.
  • Efficiency: Despite the overhead of 3D modeling, the system achieves a processing speed of 19.3 fps on an NVIDIA 2080Ti, making it viable for near real-time applications.

Limitations and Future Work

The primary bottleneck of this approach is its dependence on the accuracy of the initial 3DMM fitting. If the 3D model estimation is incorrect—common in extreme profile views or low-light conditions—the resulting UV mapping will be distorted, leading to artifacts in the final output. Future research may focus on improving the robustness of 3D face reconstruction under heavy occlusion and scaling the method to handle 4K resolution video, where current UV-mapping resolutions may struggle to preserve fine-grained skin details.

Summary updated

Abstract

This paper addresses the problem of face video inpainting. Existing video inpainting methods target primarily at natural scenes with repetitive patterns. They do not make use of any prior knowledge of the face to help retrieve correspondences for the corrupted face. They therefore only achieve sub-optimal results, particularly for faces under large pose and expression variations where face components appear very differently across frames. In this paper, we propose a two-stage deep learning method for face video inpainting. We employ 3DMM as our 3D face prior to transform a face between the image space and the UV (texture) space. In Stage I, we perform face inpainting in the UV space. This helps to largely remove the influence of face poses and expressions and makes the learning task much easier with well aligned face features. We introduce a frame-wise attention module to fully exploit correspondences in neighboring frames to assist the inpainting task. In Stage II, we transform the inpainted face regions back to the image space and perform face video refinement that inpaints any background regions not covered in Stage I and also refines the inpainted face regions. Extensive experiments have been carried out which show our method can significantly outperform methods based merely on 2D information, especially for faces under large pose and expression variations.
BibTeX
@article{yang2021deep,
  author = {Wenqi Yang and Zhenfang Chen and Chaofeng Chen and Guanying Chen and Kwan-Yee K. Wong},
  title = {{Deep Face Video Inpainting via UV Mapping}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  doi = {10.1109/tip.2023.3240835},
}
Figure