Out-of-boundary View Synthesis Towards Full-Frame Video Stabilization

Yufei Xu, Jing Zhang, Dacheng Tao

TL;DR

This paper introduces a novel "Out-of-boundary View Synthesis" (OVS) method to address the inherent cropping issue in traditional video stabilization techniques. By intelligently extrapolating missing visual information from neighboring frames, OVS enables warping-based stabilizers to achieve near full-frame stabilization, significantly improving both visual quality and stability without sacrificing cropping ratio.

What Problem Does This Paper Address?

Video stabilization aims to remove unwanted camera jitter from shaky footage, enhancing visual experience. A common approach involves warping frames to align them with a smoothed camera trajectory. However, this warping process often requires pixels that lie outside the current frame's boundary. To maintain visual consistency, these "out-of-boundary" regions are typically cropped, leading to a reduction in the effective frame size, altered aspect ratios, and sometimes amplified jitter. Existing methods often compromise between stability and cropping ratio. This paper tackles the challenge of achieving full-frame stabilization by synthesizing these missing out-of-boundary views, thus eliminating the need for aggressive cropping.

Methods

The proposed Out-of-boundary View Synthesis (OVS) method employs a two-stage coarse-to-fine approach to synthesize the missing visual information.

  • Coarse Alignment Stage: This stage uses a motion estimation module (specifically, the one from DUT) to roughly align adjacent frames with a reference frame. It identifies keypoints, estimates their motion, and then uses a grid-based homography estimation to handle large camera movements and dynamic objects. This stage generates initial aligned frames and corresponding masks.
  • Fine Alignment Stage: This stage refines the alignment. It first estimates the optical flow between the roughly aligned frame and the reference frame within the known boundary. Crucially, it then uses an "affinity propagation" mechanism to extend this optical flow information to the out-of-boundary regions. This propagation is guided by an affinity kernel estimated by a dedicated encoder-decoder network that leverages color and structural information. The extrapolated optical flow is then used to synthesize the out-of-boundary pixels.

This coarse-to-fine process is applied iteratively to gradually align distant frames, expanding the synthesized out-of-boundary view. The OVS method is designed as a plug-and-play module that can be integrated into existing warping-based video stabilizers.

Datasets

  • Training and Validation: The DeepStab dataset was used for training and validation.
  • Evaluation: The NUS dataset was used for evaluating the performance of the proposed OVS method.

Key Insights and Ideas

  • Extrapolation over Interpolation: The paper argues that for video stabilization, extrapolating out-of-boundary views before warping is more effective than interpolating missing frames after warping, as the latter can lead to permanent loss of valuable content.
  • Spatial Coherence for Extrapolation: The core idea is to leverage the spatial coherence of video content (both within a frame and across adjacent frames) to predict what lies beyond the current frame's boundary.
  • Coarse-to-Fine Alignment: The two-stage alignment process is key. The coarse stage handles large movements, while the fine stage refines alignment and uses affinity propagation to intelligently fill in the missing areas.
  • Plug-and-Play Module: The OVS method is designed to be easily integrated with existing stabilization algorithms, demonstrating its versatility and practical applicability.

Key Results

  • Improved Cropping Ratio: OVS significantly improves the cropping ratio for all tested warping-based methods, with some achieving ratios close to 1 (near full-frame). For instance, DUT+OVS achieved a cropping ratio of 0.967, and a further refinement led to 0.999.
  • Enhanced Stability and Reduced Distortion: By reducing cropping and resizing, OVS also leads to improved stability and reduced distortion in the stabilized videos.
  • Quantitative Improvements: Table 1 shows consistent improvements across multiple metrics (cropping, distortion, stability) when OVS is integrated with methods like PWStabNet, Yu et al., StabNet, MeshFlow, and DUT. For example, DUT+OVS improved the cropping ratio from 0.867 to 0.967 and distortion from 0.895 to 0.926.
  • Qualitative Improvements: Subjective comparisons (Figure 4 and 5) demonstrate that OVS effectively mitigates ghost artifacts and content loss, producing visually more appealing stabilized videos compared to methods without OVS.

Limitations and Future Work

  • Visible Seams: The authors acknowledge that due to variations in adjacent frames (illumination, dynamic objects, noise), some visible seams might still appear in the stabilized frames.
  • Refinement of Seams: They suggest that these discontinuities could potentially be refined using dedicated encoder-decoder networks.
  • Future Work: The authors plan to devise a more effective end-to-end model for improved out-of-boundary view synthesis and video stabilization. They also note that while PWCNet and their simple encoder-decoder network are effective, they are not the primary contribution, and future work could explore more advanced architectures for these components.

Summary updated

Abstract

Warping-based video stabilizers smooth camera trajectory by constraining each pixel's displacement and warp stabilized frames from unstable ones accordingly. However, since the view outside the boundary is not available during warping, the resulting holes around the boundary of the stabilized frame must be discarded (i.e., cropping) to maintain visual consistency, and thus does leads to a tradeoff between stability and cropping ratio. In this paper, we make a first attempt to address this issue by proposing a new Out-of-boundary View Synthesis (OVS) method. By the nature of spatial coherence between adjacent frames and within each frame, OVS extrapolates the out-of-boundary view by aligning adjacent frames to each reference one. Technically, it first calculates the optical flow and propagates it to the outer boundary region according to the affinity, and then warps pixels accordingly. OVS can be integrated into existing warping-based stabilizers as a plug-and-play module to significantly improve the cropping ratio of the stabilized results. In addition, stability is improved because the jitter amplification effect caused by cropping and resizing is reduced. Experimental results on the NUS benchmark show that OVS can improve the performance of five representative state-of-the-art methods in terms of objective metrics and subjective visual quality. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
BibTeX
@inproceedings{xu2021out,
  author = {Yufei Xu and Jing Zhang and Dacheng Tao},
  title = {{Out-of-boundary View Synthesis Towards Full-Frame Video Stabilization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00480},
}
Figure