Flow-Guided Diffusion for Video Inpainting

Bohai Gu, Yongsheng Yu, Heng Fan, Libo Zhang

Abstract

Video inpainting has been challenged by complex scenarios like large movements and low-light conditions. Current methods, including emerging diffusion models, face limitations in quality and efficiency. This paper introduces the Flow-Guided Diffusion model for Video Inpainting (FGDVI), a novel approach that significantly enhances temporal consistency and inpainting quality via reusing an off-the-shelf image generation diffusion model. We employ optical flow for precise one-step latent propagation and introduces a model-agnostic flow-guided latent interpolation technique. This technique expedites denoising, seamlessly integrating with any Video Diffusion Model (VDM) without additional training. Our FGDVI demonstrates a remarkable 10% improvement in flow warping error E_warp over existing state-of-the-art methods. Our comprehensive experiments validate superior performance of FGDVI, offering a promising direction for advanced video inpainting. The code and detailed results will be publicly available in https://github.com/NevSNev/FGDVI.
BibTeX
@misc{gu2023flow,
  author = {Bohai Gu and Yongsheng Yu and Heng Fan and Libo Zhang},
  title = {{Flow-Guided Diffusion for Video Inpainting}},
  year = {2023},
  eprint = {2311.15368},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.15368},
}

Loading related content...

Figure