Abstract
Video inpainting is an important technique for a wide variety of applications from video content editing to video restoration. Early approaches follow image inpainting paradigms, but are challenged by complex camera motion and non-rigid deformations. To address these challenges flow-guided propagation techniques have been proposed. However, computation of flow is non-trivial for unobserved regions and propagation across a whole video sequence is computationally demanding. In contrast, in this paper, we propose a video inpainting algorithm based on proposals: we use 3D convolutions to obtain an initial inpainting estimate which is subsequently refined by fusing a generated set of proposals. Different from existing approaches for video inpainting, and inspired by well-explored mechanisms for object detection, we argue that proposals provide a rich source of information that permits combining similarly looking patches that may be spatially and temporally far from the region to be inpainted. We validate the effectiveness of our method on the challenging YouTube VOS and DAVIS datasets using different settings and demonstrate results outperforming state-of-the-art on standard metrics.
Introduction
Video inpainting—the process of filling in missing regions or removing objects from a video—is a fundamental task in video editing and restoration. While image inpainting has seen significant progress, extending these techniques to video introduces the critical requirement of temporal consistency. Early methods that applied image inpainting frame-by-frame often suffered from severe flickering and lack of coherence.
Recent state-of-the-art approaches have turned to optical flow to propagate known pixels into missing regions. However, this introduces a circular dependency: accurate optical flow is difficult to estimate for regions that are missing or occluded. Furthermore, the iterative nature of flow propagation across long sequences is computationally expensive, limiting its utility for real-time or mobile applications.
Methodology
The proposed framework shifts the paradigm from local flow-based propagation to a global proposal-based matching system. The pipeline consists of three main stages:
- Coarse 3D Inpainting: A 3D Convolutional Neural Network (CNN) with gated convolutions generates an initial, coarse estimate of the missing pixels. This stage provides a structural "blueprint" of what the missing content should look like.
- Proposal Generation: Inspired by object detection, the model identifies candidate patches (proposals) from observed pixels throughout the entire video. By matching features from the coarse estimate against all available pixels, the system can find high-quality matches that may be spatially or temporally distant—effectively "teleporting" textures from frames far removed from the current one.
- Proposal Fusion: A learned attention-based classifier decides how to blend the top-$k$ proposals. This fusion ensures that the final output maintains sharp textures and remains consistent with the surrounding environment.
Experiments and Results
The model was evaluated on the YouTube VOS and DAVIS datasets. These benchmarks provide a diverse set of challenges, including complex motions and high-resolution object removal tasks.
Performance and Efficiency
The proposal-based approach demonstrates a significant leap in efficiency. On the DAVIS dataset, the method achieves an inference speed of 0.69 seconds per frame, representing a 55x speedup over the flow-guided FGI method, which requires 37.63 seconds per frame.
Qualitative Evaluation
Quantitatively, the method outperforms previous state-of-the-art models like OPN and VINet in terms of PSNR and SSIM. Qualitatively, the system produces significantly fewer artifacts in scenes with non-rigid deformations (e.g., a person dancing) or cluttered backgrounds.
Conclusion
By replacing local flow propagation with a global proposal-based matching mechanism, this work addresses the core challenges of efficiency and temporal consistency in video inpainting. While the system currently faces difficulties with extremely thin structures when the missing region ratio is high, it sets a new benchmark for speed and accuracy in video completion. Future research will focus on improving the fusion of proposals to better handle fine-scale geometric details.
Summary updated