Deep Iterative Frame Interpolation for Full-frame Video Stabilization

Jinsoo Choi, In So Kweon

TL;DR

This paper introduces DIFRINT, a novel deep learning approach for video stabilization that uses iterative frame interpolation to achieve full-frame stabilization without cropping. By generating intermediate frames, it significantly reduces jitter and distortion, offering a compelling alternative to traditional methods that rely on cropping.

What Problem Does This Paper Address?

Video stabilization is crucial for producing high-quality videos, especially from shaky sources like handheld cameras. Existing methods, while effective, often suffer from a major drawback: they require cropping the video frames to hide the distorted or missing content that arises from compensating for camera motion. This cropping leads to a loss of original image information and an undesirable zoom-in effect. Furthermore, many advanced methods require paired "shaky" and "stable" video data for supervised training, which can be difficult to obtain. This paper aims to address these limitations by developing a video stabilization technique that is full-frame, produces low distortion, and can be trained in an unsupervised manner.

Methods

The core of DIFRINT lies in its innovative use of deep frame interpolation for stabilization. Instead of directly estimating camera motion and then warping, the method generates "in-between" frames. The process involves:

  1. Bidirectional Warping: Given two consecutive frames ($f_i$ and $f_{i+1}$), optical flow is computed to warp each frame towards the other. This creates two warped frames, $f_w$ and $f_{w'}$, representing intermediate positions.
  2. U-Net for Intermediate Frame: These warped frames are fed into a U-Net architecture, which learns to generate an initial intermediate frame, $f_{int}$. This U-Net is designed to combine information at different scales, crucial for generating plausible intermediate content.
  3. ResNet for Final Frame: To further refine the result and prevent blur accumulation, the original frame $f_i$ is warped towards $f_{int}$ and then, along with $f_{int}$, fed into a ResNet architecture. This ResNet produces the final stabilized frame, $f_i$.
  4. Iterative Stabilization: The key to DIFRINT's effectiveness is its iterative application. The output of one stabilization step can be used as an input for the next, progressively smoothing out jitter. This iterative process allows for a more global stabilization effect, as frames are influenced by farther-away frames in subsequent iterations.
  5. Unsupervised Training: A significant contribution is the unsupervised training strategy. Since ground truth intermediate frames don't exist, the method trains by reconstructing a "pseudo-ground truth" frame ($f_s$), which is a slightly scaled and translated version of the original frame. This allows the network to learn to generate stable frames without needing paired stable video data.

Datasets

  • Training: The DAVIS dataset was used for training. This dataset is noted for containing diverse and dynamic scenes.
  • Evaluation: The authors evaluated their method on 25 publicly available videos, comparing against 13 baselines. They also conducted user studies using videos from 6 categories: Regular, Crowd, Parallax, Quick Rotation, Running, and Zooming.

Key Insights and Ideas

  • Frame Interpolation as Stabilization: The fundamental insight is reframing video stabilization not just as motion compensation but as a problem of generating missing temporal information through interpolation. This elegantly sidesteps the need for cropping.
  • Iterative Refinement for Global Stability: The iterative nature of the process is crucial. It allows the stabilization to propagate beyond immediate neighbors, leading to a more globally consistent and smooth output, as opposed to local jitter reduction.
  • Unsupervised Learning for Practicality: The development of an unsupervised training scheme is a major practical advantage, removing the bottleneck of acquiring paired shaky/stable video datasets.
  • Full-Frame Generation: By generating content rather than discarding it, the method achieves full-frame stabilization, preserving all original image information.

Key Results

  • Quantitative: DIFRINT achieved cropping ratios of 1.0 (no cropping) and distortion values close to 1.0 across various test videos. It also demonstrated superior stability scores compared to many state-of-the-art methods.
  • Qualitative: Visual comparisons showed that DIFRINT generates stabilized frames without cropping, unlike other methods that exhibit significant cropping. The generated unseen regions at boundaries were visually appealing and preserved content.
  • Speed: The method runs at near real-time speeds of 15 fps on a single GPU, making it practical for many applications.
  • User Study: In a user preference study, DIFRINT was favored by a majority of participants across all tested video categories, indicating a preference for its full-frame, low-distortion output over commercial alternatives.

Limitations and Future Work

  • Blur Artifacts on Severe Shakes: The paper acknowledges that severe camera shakes can still lead to blur artifacts at image boundaries, especially when generating large unseen regions.
  • Linear Interpolation Assumption: The current method applies linear interpolation to all motion types without discrimination.
  • Future Directions: The authors suggest dynamically determining motion types within video segments and applying stabilization accordingly. They also propose exploring the use of multiple neighboring frames to reconstruct larger unseen regions more effectively, potentially addressing the blur issue in extreme cases.

Summary updated

Abstract

Video stabilization is a fundamental and important technique for higher quality videos. Prior works have extensively explored video stabilization, but most of them involve cropping of the frame boundaries and introduce moderate levels of distortion. We present a novel deep approach to video stabilization that can generate video frames without cropping and low distortion. The proposed framework utilizes frame interpolation techniques to generate in between frames, leading to reduced inter-frame jitter. Once applied in an iterative fashion, the stabilization effect becomes stronger. A major advantage is that our framework is end-to-end trainable in an unsupervised manner. In addition, our method is able to run in near real-time (15 fps). To the best of our knowledge, this is the first work to propose an unsupervised deep approach to full-frame video stabilization. We show the advantages of our method through quantitative and qualitative evaluations comparing to the state-of-the-art methods.
BibTeX
@article{choi2020deep,
  author = {Jinsoo Choi and In So Kweon},
  title = {{Deep Iterative Frame Interpolation for Full-frame Video Stabilization}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  url = {https://openreview.net/forum?id=PCoE4gr_iX},
}
Figure