BVINet: Unlocking Blind Video Inpainting with Zero Annotations

Zhiliang Wu, Kerui Chen, Kun Li, Hehe Fan, Yi Yang

Abstract

Video inpainting aims to fill in corrupted regions of the video with plausible contents. Existing methods generally assume that the locations of corrupted regions are known, focusing primarily on the "how to inpaint". This reliance necessitates manual annotation of the corrupted regions using binary masks to indicate "where to inpaint". However, the annotation of these masks is labor-intensive and expensive, limiting the practicality of current methods. In this paper, we expect to relax this assumption by defining a new blind video inpainting setting, enabling the networks to learn the mapping from corrupted video to inpainted result directly, eliminating the need for corrupted region annotations. Specifically, we propose an end-to-end blind video inpainting network (BVINet) to address both "where to inpaint" and "how to inpaint" simultaneously. On the one hand, BVINet can predict the masks of corrupted regions by detecting semantic-discontinuous regions of the frame and utilizing temporal consistency prior of the video. On the other hand, the predicted masks are incorporated into the BVINet, allowing it to capture valid context information from uncorrupted regions to fill in corrupted ones. Besides, we introduce a consistency loss to regularize the training parameters of BVINet. In this way, mask prediction and video completion mutually constrain each other, thereby maximizing the overall performance of the trained model. Furthermore, we customize a dataset consisting of synthetic corrupted videos, real-world corrupted videos, and their corresponding completed videos. This dataset serves as a valuable resource for advancing blind video inpainting research. Extensive experimental results demonstrate the effectiveness and superiority of our method.

Introduction

Traditional video inpainting is a "non-blind" task, meaning the algorithm requires a human-provided binary mask for every single frame to identify which pixels require restoration. While effective, this requirement creates a massive bottleneck: manually annotating high-resolution, high-frame-rate video is labor-intensive and often inaccurate due to blurred boundaries between corrupted and clean regions.

Comparison of non-blind and blind video inpainting pipelines.

To overcome this, we introduce blind video inpainting, where the model must learn to map a corrupted video directly to a clean one. This requires the network to solve two problems simultaneously: localization ("where to inpaint") and synthesis ("how to inpaint"). The primary challenge lies in maintaining temporal consistency—preventing the "flicker" effect common when processing video frames individually—without the guidance of ground-truth masks during inference.

Methodology: The BVINet Framework

BVINet is the first end-to-end framework designed for blind video inpainting. It consists of two primary components that work in tandem: the Mask Prediction Network (MPNet) and the Video Completion Network (VCNet).

The overall architecture of BVINet, including the Mask Prediction Network and Video Completion Network.

1. Mask Prediction Network (MPNet)

This module identifies corrupted regions by detecting semantic discontinuities and utilizing temporal consistency. It employs a short-term prediction module built on an encoder-decoder architecture with Discrete Wavelet Transforms (DWT) to improve noise immunity. A long-term refinement module then analyzes multiple frames to ensure the predicted masks are stable over time, preventing temporal artifacts.

2. Video Completion Network (VCNet)

Once the masks are predicted, VCNet fills them in using a Wavelet Sparse Transformer. This component uses DWT to separate features into low-frequency (clean) and high-frequency (noisy) components. It then applies a dual-branch attention mechanism:

  • Dense Self-Attention (DSA): Captures broad spatial-temporal context.
  • Sparse Self-Attention (SSA): Uses a ReLU-based mask to filter out irrelevant or noisy features, preventing the blurry results often seen in traditional transformers.

3. Consistency Loss

A novel consistency loss function ties the two networks together. It ensures that the difference between the corrupted input and the restored output matches the predicted mask. This forces both networks to "agree" on the location of damage, allowing mask prediction and video completion to mutually constrain and improve each other.

Datasets and Training

Existing datasets often use simple black masks that are too easy for a network to identify. To force the model to learn deeper semantics, we developed a Synthesized Dataset of 2,400 clips where corruptions are patches from other natural images, blended with iterative Gaussian smoothing. We also introduced a Real-World Dataset of 1,250 clips focused on "bullet removal" (removing text/graphics overlays from news or social media videos).

Key Results

BVINet achieves state-of-the-art performance while remaining remarkably efficient. On the YouTube-VOS dataset, it reached a PSNR of 30.528, significantly outperforming other blind inpainting baselines.

In terms of computational cost, BVINet is lean, using only 396.35G FLOPs with an inference time of 0.21s per 20-frame sequence. This makes it faster and more efficient than many non-blind state-of-the-art models like STTN or FuseFormer. Qualitative results show that the model effectively removes complex text overlays and scratches in real-world footage without leaving ghosting artifacts.

Conclusion

BVINet represents a significant step forward in making video restoration practical for real-world applications. By eliminating the need for manual annotations through a coupled mask-prediction and completion architecture, it enables high-quality, efficient video repair at scale. Future work will explore extending this zero-annotation approach to internal object removal, where corrupted regions are semantically consistent with the background.

Summary updated

Abstract

Video inpainting aims to fill in corrupted regions of the video with plausible contents. Existing methods generally assume that the locations of corrupted regions are known, focusing primarily on the "how to inpaint". This reliance necessitates manual annotation of the corrupted regions using binary masks to indicate "whereto inpaint". However, the annotation of these masks is labor-intensive and expensive, limiting the practicality of current methods. In this paper, we expect to relax this assumption by defining a new blind video inpainting setting, enabling the networks to learn the mapping from corrupted video to inpainted result directly, eliminating the need of corrupted region annotations. Specifically, we propose an end-to-end blind video inpainting network (BVINet) to address both "where to inpaint" and "how to inpaint" simultaneously. On the one hand, BVINet can predict the masks of corrupted regions by detecting semantic-discontinuous regions of the frame and utilizing temporal consistency prior of the video. On the other hand, the predicted masks are incorporated into the BVINet, allowing it to capture valid context information from uncorrupted regions to fill in corrupted ones. Besides, we introduce a consistency loss to regularize the training parameters of BVINet. In this way, mask prediction and video completion mutually constrain each other, thereby maximizing the overall performance of the trained model. Furthermore, we customize a dataset consisting of synthetic corrupted videos, real-world corrupted videos, and their corresponding completed videos. This dataset serves as a valuable resource for advancing blind video inpainting research. Extensive experimental results demonstrate the effectiveness and superiority of our method.
BibTeX
@inproceedings{wu2025bvinet,
  author = {Zhiliang Wu and Kerui Chen and Kun Li and Hehe Fan and Yi Yang},
  title = {{BVINet: Unlocking Blind Video Inpainting with Zero Annotations}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure