CIRI: Curricular Inactivation for Residue-aware One-shot Video Inpainting

Weiying Zheng, Cheng Xu, Xuemiao Xu, Wenxi Liu, Shengfeng He

Abstract

Video inpainting typically requires precise per-frame masks, a labor-intensive process that limits its accessibility. CIRI (Curricular Inactivation for Residue-aware one-shot video Inpainting) addresses this bottleneck by enabling high-quality object removal using only a single annotated frame. It introduces a feature-level "target inactivation" strategy that is more error-tolerant than traditional image-space masking, alongside an online residue removal process that iteratively cleans up artifacts during inference. Experimental results demonstrate that CIRI significantly improves inpainting quality and temporal consistency across standard benchmarks.

Introduction

Video inpainting—the process of seamlessly filling in missing regions or removing objects—is a staple of modern post-production. However, state-of-the-art models often require a precise mask for every frame, which is a Herculean task for high-frame-rate videos. While "one-shot" methods exist, they frequently fail because slight inaccuracies in mask propagation lead to "residues" (ghostly remnants of the object) or the destruction of useful background information. These errors compound over time, resulting in flickering and blurry artifacts.

Methodology

CIRI is a "plug-and-play" framework designed to enhance existing inpainting models. It shifts the masking process from the image space to the latent feature space through two primary innovations:

1. Curricular Target Inactivation

Instead of using a hard mask to zero out pixels in the image space, CIRI "inactivates" multi-scale feature responses in the latent space. This approach is inherently more error-tolerant because high-level features have larger receptive fields; even if a mask is slightly inaccurate, the network can still leverage surrounding context to understand what needs to be filled. To make the model robust to real-world mask propagation errors, the authors employ Dual-Curriculum Learning, training the model in stages that progress from perfect ground-truth masks to noisy, predicted ones.

2. Online Residue Removal

To address inevitable artifacts, CIRI includes a module that runs during the inference phase. It utilizes a modified object tracking network to detect "inpainting residues"—areas that look like the original object but should not be present. It then performs an iterative "detect-and-refine" process, fine-tuning the mask prediction on-the-fly to scrub away these leftover pixels.

The CIRI framework architecture, featuring Curricular Target Learning, the Video Inpainting Network with Target Inactivation, and the Online Residue Removal module.

Key Insights

  • Feature-level Inactivation: Shifting from masking pixels to inactivating features allows the model to be "fuzzy" about boundaries, which is crucial when mask propagation is imperfect.
  • Inference-time Intelligence: By treating inference as an active process with online residue removal, CIRI can correct its own mistakes after the initial pass.
  • Universal Compatibility: CIRI acts as a wrapper for various backbones (e.g., STTN, FuseFormer, E2FGVI), consistently boosting their performance in one-shot scenarios.

Results and Evaluation

CIRI demonstrated significant quantitative and qualitative leads over existing one-shot methods. On synthetic datasets, it improved local PSNR (e.g., from 28.11 to 29.01 for the E2FGVI backbone) and significantly reduced VFID scores, indicating smoother, more natural results. Qualitative comparisons show that CIRI effectively removes residues that other methods leave behind, maintaining high temporal coherence.

Qualitative comparison of CIRI against state-of-the-art methods (STTN, FuseFormer, E2FGVI, IIVI). CIRI demonstrates superior residue removal and temporal consistency across various challenging scenarios.

Conclusion and Limitations

CIRI represents a significant advancement in one-shot video inpainting, making high-quality video editing significantly less laborious. Its primary limitation is the computational overhead introduced by the online residue removal step, which increases inference time. Additionally, the system remains dependent on the quality of the underlying mask propagation; extreme viewpoint changes can still lead to tracking failures. Future work will focus on making mask propagation more robust to such transformations.

Summary updated

Abstract

Video inpainting aims at filling in missing regions of a video. However, when dealing with dynamic scenes with camera or object movements, annotating the inpainting target becomes laborious and impractical. In this paper, we resolve the one-shot video inpainting problem in which only one annotated first frame is provided. A naive solution is to propagate the initial target to the other frames with techniques like object tracking. In this context, the main obstacles are the unreliable propagation and the partially inpainted artifacts due to the inaccurate mask. For the former problem, we propose curricular inactivation to replace the hard masking mechanism for indicating the in-painting target, which is robust to erroneous predictions in long-term video inpainting. For the latter, we explore the properties of inpainting residue and present an online residue removal method in an iterative detect-and-refine manner. Extensive experiments on several real-world datasets demonstrate the quantitative and qualitative superiorities of our proposed method in one-shot video inpainting. More importantly, our method is extremely flexible that can be integrated with arbitrary traditional inpainting models, activating them to perform the reliable one-shot video inpainting task. Video demonstrations can be found in our supplement, and our code can be found at https://github.com/Arise-zwy/CIRI.
BibTeX
@inproceedings{zheng2023ciri,
  author = {Weiying Zheng and Cheng Xu and Xuemiao Xu and Wenxi Liu and Shengfeng He},
  title = {{CIRI: Curricular Inactivation for Residue-aware One-shot Video Inpainting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01196},
}
Figure