Introduction
Video inpainting—the process of filling missing regions or "holes" in a video sequence—presents a unique challenge in computer vision. It requires maintaining high-resolution spatial details within individual frames while ensuring perfect temporal consistency across the entire sequence. Traditional approaches often struggle with this balance: frame-by-frame 2D image inpainting frequently results in temporal "jitter" or "flickering," while 3D convolutional approaches are computationally expensive and often limited to low-resolution, blurry outputs due to memory constraints.
This paper introduces a novel two-stage deep learning architecture that decouples the learning of temporal structure from spatial details. By using a low-resolution 3D CNN to "sketch" motion and a high-resolution 2D CNN to "paint" fine details, the method produces videos that are both sharp and temporally coherent.
Proposed Architecture: The CombCN
The core of the proposed method is a hybrid architecture consisting of two sub-networks trained end-to-end: the 3D Completion Network (3DCN) and the Combined Completion Network (CombCN).
- 3D Completion Network (3DCN): This network treats the video as a 3D volume but operates on a down-sampled version to manage memory usage. Its primary role is to capture the "temporal structure"—understanding how objects move and how missing regions should evolve over time. It generates a low-resolution, coarse version of the completed video.
- Combined Completion Network (CombCN): This is a 2D CNN that processes frames at their original high resolution. To ensure temporal consistency, it does not operate in isolation; instead, it fuses feature maps from the 3DCN into its own layers (specifically the 2nd and 15th layers). This allows the 3DCN to act as a temporal guide, providing a global motion context for the high-resolution spatial synthesis.
Joint training allows the spatial detail loss to backpropagate into the 3D temporal network, forcing the 3DCN to learn more precise motion structures that support high-fidelity reconstruction.
Experimental Setup
The model was evaluated on three distinct datasets to test its versatility across structured and unstructured content:
- FaceForensics & 300VW: These datasets contain video clips of human faces, testing the model's ability to reconstruct rigid but expressive semantic structures.
- Caltech Pedestrian Dataset: This dataset consists of urban traffic video. It is significantly more challenging due to the lack of a fixed semantic structure and the presence of complex, unstructured motion from vehicles and pedestrians.
Results and Discussion
The proposed method consistently outperformed "pure 2D" and "pure 3D" baselines both quantitatively and qualitatively.
Quantitative Performance
The model achieved the lowest $l_1$ error across all tested datasets. Notably, on the challenging Caltech dataset, the proposed method achieved an $l_1$ loss of 9.56, significantly lower than the 3D-only (11.91) and 2D-only (11.16) versions.
Qualitative Analysis
Qualitative results highlight the strength of the temporal guidance. In facial videos where the mouth was obscured, 2D methods often failed to maintain consistency, leading to "disappearing mouth" artifacts. The proposed method maintained a realistic and consistent mouth shape throughout the sequence. Furthermore, the system demonstrated robustness to "random holes" (scribble-like masks that change per frame) by effectively leveraging information from surrounding unmasked frames via the 3DCN.
Conclusion and Future Work
The decoupling of structure and detail allows the model to bypass hardware limitations associated with 3D convolutions while maintaining high-resolution output. However, limitations remain. Large or extremely fast motion can exceed the receptive field of the 3D CNN, leading to distortions. Additionally, the current reliance on $l_1$ loss can result in overly smooth textures. Future work may involve integrating Generative Adversarial Networks (GANs) for more vivid textures and exploring optical flow or LSTM-based architectures for longer-range temporal dependencies.
Summary updated