Video Inpainting by Jointly Learning Temporal Structure and Spatial Details.

Chuan Wang, Haibin Huang, Xiaoguang Han, Jue Wang

Introduction

Video inpainting—the process of filling missing regions or "holes" in a video sequence—presents a unique challenge in computer vision. It requires maintaining high-resolution spatial details within individual frames while ensuring perfect temporal consistency across the entire sequence. Traditional approaches often struggle with this balance: frame-by-frame 2D image inpainting frequently results in temporal "jitter" or "flickering," while 3D convolutional approaches are computationally expensive and often limited to low-resolution, blurry outputs due to memory constraints.

This paper introduces a novel two-stage deep learning architecture that decouples the learning of temporal structure from spatial details. By using a low-resolution 3D CNN to "sketch" motion and a high-resolution 2D CNN to "paint" fine details, the method produces videos that are both sharp and temporally coherent.

Proposed Architecture: The CombCN

The core of the proposed method is a hybrid architecture consisting of two sub-networks trained end-to-end: the 3D Completion Network (3DCN) and the Combined Completion Network (CombCN).

The proposed architecture consists of a 3DCN for temporal structure and a CombCN for high-resolution spatial details.
  1. 3D Completion Network (3DCN): This network treats the video as a 3D volume but operates on a down-sampled version to manage memory usage. Its primary role is to capture the "temporal structure"—understanding how objects move and how missing regions should evolve over time. It generates a low-resolution, coarse version of the completed video.
  2. Combined Completion Network (CombCN): This is a 2D CNN that processes frames at their original high resolution. To ensure temporal consistency, it does not operate in isolation; instead, it fuses feature maps from the 3DCN into its own layers (specifically the 2nd and 15th layers). This allows the 3DCN to act as a temporal guide, providing a global motion context for the high-resolution spatial synthesis.

Joint training allows the spatial detail loss to backpropagate into the 3D temporal network, forcing the 3DCN to learn more precise motion structures that support high-fidelity reconstruction.

Experimental Setup

The model was evaluated on three distinct datasets to test its versatility across structured and unstructured content:

  • FaceForensics & 300VW: These datasets contain video clips of human faces, testing the model's ability to reconstruct rigid but expressive semantic structures.
  • Caltech Pedestrian Dataset: This dataset consists of urban traffic video. It is significantly more challenging due to the lack of a fixed semantic structure and the presence of complex, unstructured motion from vehicles and pedestrians.

Results and Discussion

The proposed method consistently outperformed "pure 2D" and "pure 3D" baselines both quantitatively and qualitatively.

Qualitative comparison of the proposed method (CombCN) against 3DCN and 2DCN baselines across face and traffic datasets.

Quantitative Performance

The model achieved the lowest $l_1$ error across all tested datasets. Notably, on the challenging Caltech dataset, the proposed method achieved an $l_1$ loss of 9.56, significantly lower than the 3D-only (11.91) and 2D-only (11.16) versions.

Qualitative Analysis

Qualitative results highlight the strength of the temporal guidance. In facial videos where the mouth was obscured, 2D methods often failed to maintain consistency, leading to "disappearing mouth" artifacts. The proposed method maintained a realistic and consistent mouth shape throughout the sequence. Furthermore, the system demonstrated robustness to "random holes" (scribble-like masks that change per frame) by effectively leveraging information from surrounding unmasked frames via the 3DCN.

Conclusion and Future Work

The decoupling of structure and detail allows the model to bypass hardware limitations associated with 3D convolutions while maintaining high-resolution output. However, limitations remain. Large or extremely fast motion can exceed the receptive field of the 3D CNN, leading to distortions. Additionally, the current reliance on $l_1$ loss can result in overly smooth textures. Future work may involve integrating Generative Adversarial Networks (GANs) for more vivid textures and exploring optical flow or LSTM-based architectures for longer-range temporal dependencies.

Summary updated

Abstract

We present a new data-driven video inpainting method for recovering missing regions of video frames. A novel deep learning architecture is proposed which contains two subnetworks: a temporal structure inference network and a spatial detail recovering network. The temporal structure inference network is built upon a 3D fully convolutional architecture: it only learns to complete a low-resolution video volume given the expensive computational cost of 3D convolution. The low resolution result provides temporal guidance to the spatial detail recovering network, which performs imagebased inpainting with a 2D fully convolutional network to produce recovered video frames in their original resolution. Such two-step network design ensures both the spatial quality of each frame and the temporal coherence across frames. Our method jointly trains both sub-networks in an end-to-end manner. We provide qualitative and quantitative evaluation on three datasets, demonstrating that our method outperforms previous learning-based video inpainting methods.
BibTeX
@inproceedings{wang2019video,
  author = {Chuan Wang and Haibin Huang and Xiaoguang Han and Jue Wang},
  title = {{Video Inpainting by Jointly Learning Temporal Structure and Spatial Details.}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015232},
}
Figure