Spatial-Temporal Residual Aggregation for High Resolution Video Inpainting

Vishnu Sanjay Ramiya Srinivasan, Rui Ma, Qiang Tang, Zili Yi, Zhan Xu

TL;DR

This paper introduces STRA-Net, a novel framework for high-resolution video inpainting that addresses memory constraints by performing the core inpainting on downsampled videos and then refining the results with residual aggregation. This approach achieves state-of-the-art performance, producing temporally coherent and visually appealing inpainted videos without requiring high-resolution training data.

What Problem Does This Paper Address?

Video inpainting, the task of filling in missing regions in video frames, has numerous applications like object removal and video restoration. While deep learning methods have shown great promise, they often struggle with high-resolution videos due to significant memory and computational demands. Existing approaches typically require high-resolution training data or are limited to low-resolution inputs, leaving a gap for efficient and effective high-resolution video inpainting.

Methods

The proposed STRA-Net framework tackles high-resolution video inpainting by combining a novel spatial-temporal residual aggregation approach with a modified learning-based inpainting network.

The core of the method is the Spatial-Temporal Aggregation Network (STA-Net), which is trained on downsampled low-resolution videos. STA-Net is inspired by the Copy-Paste Network (CPNet) but introduces several improvements:

  • Joint Alignment Network: This module combines affine transformation and optical flow for more robust image alignment between the target and reference frames, capturing both global and local motion.
  • Multi-scale Temporal Aggregation: This module aggregates temporal features across multiple scales, enhancing the transfer of information from neighboring frames.
  • Multi-scale Spatial Aggregation: This module refines the inpainted regions using contextual information within the target image itself, similar to techniques used in high-resolution image inpainting.

After STA-Net generates an initial low-resolution inpainting, the Spatial-Temporal Residual Aggregation step refines the result to high resolution. This involves:

  1. Extracting Residuals: High-frequency details (residuals) are extracted from both the original high-resolution frames and the downsampled STA-Net output.
  2. Aligning Residuals: The temporal residuals are aligned to the target frame using the learned affine and flow transformations from STA-Net.
  3. Aggregating Residuals:
    • Temporal Residual Aggregation: High-frequency details from aligned temporal residuals are transferred to the upsampled STA-Net output using temporal attention scores.
  4. Spatial Residual Aggregation: Similar to image inpainting methods like HiFill, spatial residuals from surrounding patches are aggregated to further refine the inpainted region.

This residual aggregation process allows the model to leverage high-frequency details from the original high-resolution frames without directly processing them through the entire deep network, thus mitigating memory constraints.

Datasets

  • Training Data: A synthesized dataset was created by compositing object/human masks onto 1.8M images from Places [47], Pascal VOC [10], MS COCO [27], and Open Image [24]. Additionally, 33.7K natural scene video clips were processed. All training data is resized to 512x512.
  • Testing Datasets:
    • DS+: 50 sequences from the DAVIS dataset [32] with added shadow annotations for realism. Videos are resized to 1080p.
  • HIN: 30 real-world human-focused 1080p videos.
  • Syn-DS+: A synthetic dataset generated by imposing DS+ masks onto 50 real-world videos with natural scenes and city views.

Key Insights and Ideas

  • Residuals as a Memory-Saving Trick: The core insight is that high-resolution details can be effectively transferred and integrated by focusing on the residuals (high-frequency differences) rather than the full high-resolution frames during the refinement stage. This allows the computationally intensive STA-Net to operate on low-resolution inputs.
  • Joint Alignment for Robustness: Combining affine and optical flow alignment provides a more robust approach to image registration, capturing both global scene motion and local object movements, which is crucial for accurate information transfer.
  • Multi-scale Attention for Detail: Employing multi-scale attention mechanisms in both temporal and spatial aggregation modules allows for a more refined selection and transfer of features, leading to better detail reconstruction.
  • Modular Design for Adaptability: The framework is modular, allowing STA-Net to be potentially adapted by other video inpainting methods to improve their high-resolution capabilities.

Key Results

  • Quantitative Performance: The paper reports superior quantitative results (L1, SSIM, PSNR) compared to state-of-the-art methods like CPNet, STTN, DFG, and FGVC across various resolutions, including 1728x960. For example, on the Syn-DS+ dataset at 1728x960, their method achieved an SSIM of 0.97 and PSNR of 33.14, outperforming baselines.
  • Efficiency: STRA-Net achieves competitive running times (e.g., 1.2 seconds for a 1728x960 frame) and significantly lower memory consumption (e.g., 3.7 GB for 1728x960) compared to other methods, enabling high-resolution inpainting on standard hardware.
  • Qualitative Performance: Visual results demonstrate that STRA-Net produces more temporally coherent and visually appealing inpainted videos, with fewer artifacts and better detail preservation, especially in challenging regions.
  • User Study: In a user study, STRA-Net was ranked as the best in 39.8% of cases and second best in 53.6%, outperforming CPNet and STTN and performing comparably to FGVC, while being significantly faster and more memory-efficient.

Limitations and Future Work

  • Limitations:
    • While the method is efficient, some baselines struggle to run on 1080p videos due to memory constraints, necessitating resizing for fair comparison.
  • The paper notes that for some regions, there might be a "leftover mask" where no temporal information is transferred, potentially leading to blurry pixels generated by dilated convolutions. Spatial aggregation aims to refine these.
  • Future Work: The authors suggest exploring the application of STRA-Net to other video processing tasks such as efficient HD video synthesis and animation.

Summary updated

Abstract

Recent learning-based inpainting algorithms have achieved compelling results for completing missing regions after removing undesired objects in videos. To maintain the temporal consistency among the frames, 3D spatial and temporal operations are often heavily used in the deep networks. However, these methods usually suffer from memory constraints and can only handle low resolution videos. We propose STRA-Net, a novel spatial-temporal residual aggregation framework for high resolution video inpainting. The key idea is to first learn and apply a spatial and temporal inpainting network on the downsampled low resolution videos. Then, we refine the low resolution results by aggregating the learned spatial and temporal image residuals (details) to the upsampled inpainted frames. Both the quantitative and qualitative evaluations show that we can produce more temporal-coherent and visually appealing results than the state-of-the-art methods on inpainting high resolution videos.
BibTeX
@inproceedings{srinivasan2021spatial,
  author = {Vishnu Sanjay Ramiya Srinivasan and Rui Ma and Qiang Tang and Zili Yi and Zhan Xu},
  title = {{Spatial-Temporal Residual Aggregation for High Resolution Video Inpainting}},
  booktitle = {BMVC},
  year = {2021},
}
Figure