Self-Supervised Real-time Video Stabilization

Jinsoo Choi, Jaesik Park, In So Kweon

TL;DR

Researchers have developed a new deep-learning framework for real-time video stabilization that learns entirely from "shaky" data—no expensive gimbal-stabilized ground truth required. By combining a coarse-to-fine stabilization pipeline with an intelligent margin inpainter, the system achieves 41 FPS performance while avoiding the aggressive "zoom-in" effect typical of smartphone stabilizers.

What Problem Does This Paper Address?

If you’ve ever recorded video on a smartphone, you’ve likely encountered the trade-offs of stabilization. Hardware solutions like Optical Image Stabilization (OIS) handle small jitters, while Electronic Image Stabilization (EIS) tackles larger shakes. However, EIS usually works by cropping the edges of the frame and "zooming in" to create a buffer for movement. This results in a loss of field-of-view and image quality.

Existing software-based AI solutions often fall into two traps: they are either too slow for real-time use (offline) or they require "supervised" training. Supervised training is a logistical nightmare—it requires researchers to film the same scene simultaneously with a shaky handheld camera and a perfectly stabilized gimbal rig to create "before and after" training pairs. This paper aims to break that bottleneck with a self-supervised approach that works in real-time.

Methods

The authors propose a three-stage "cascade" architecture that processes video frames sequentially:

  1. Coarse-Net (Global Stability): This module estimates rigid transformations (rotation and translation) between frames. It uses a U-Net architecture to process optical flow and applies a moving average filter to "iron out" abrupt camera jumps.
  2. Fine-Net (Detail Stability): Global rotation isn't enough to fix complex issues like parallax (where objects at different depths move differently). Fine-Net uses a tiny U-shaped network to generate a spatially smoothed optical flow map. This allows for "warping" the image to fix local instabilities without being distracted by moving objects in the foreground (like a person walking across the frame).
  3. Margin-Net (The "Anti-Zoom" Tool): Instead of cropping the messy edges created by stabilization, Margin-Net uses information from adjacent frames to inpaint the missing boundaries. This allows the final video to maintain a much wider field of view.

The Self-Supervised Secret Sauce: To train these nets without ground truth, the authors use "Learning by Random Augmentation." They take a single stable frame, apply a synthetic "shake" (random rotation/translation), and task the network with reversing that transformation. They also use temporal consistency losses to ensure that adjacent frames in a real video sequence align smoothly.

Datasets

  • Training: The model was trained on the DAVIS dataset, a standard benchmark for video object segmentation. Crucially, it was trained on the raw video without needing stabilized counterparts.
  • Evaluation: The authors tested their method on a union of 19 video clips from the Liu et al. benchmark, covering various motion types: forward/backward, side-to-side, zoom, and rolling shutter artifacts.

Key Insights and Ideas

  • 3-DoF is Enough: The authors found that a 3-Degree-of-Freedom (rotation and 2D translation) model was surprisingly more effective for stabilization than higher-complexity models (like those including scale or shear), which can sometimes introduce "wobble" artifacts.
  • Foreground Agnostic: A major "aha" moment is the Fine-Net’s ability to ignore foreground movement. By using average pooling and down-sampling, the network naturally filters out high-frequency "noise" (like a person's limbs moving) and focuses on the low-frequency global motion of the background.
  • Inpainting vs. Cropping: Most stabilizers treat the frame boundary as a hard limit. By treating the boundary as an inpainting task, the authors turn a geometric problem into a generative one, significantly reducing the "zoom-in" effect.

Key Results

  • Speed: The entire pipeline takes approximately 24.3 ms per frame, yielding a consistent 41 FPS. This is true regardless of whether the resolution is 480p or 1080p, making it viable for mobile devices.
  • Quality: In quantitative tests, the method outperformed state-of-the-art real-time stabilizers in three key metrics: cropping ratio (keeping more of the original image), distortion value (fewer "wavy" artifacts), and stability score.
  • Visuals: Qualitatively, the results show much less blur and a significantly wider field of view compared to commercial products like Adobe Premiere’s Warp Stabilizer or standard smartphone OIS+EIS.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain open for exploration:

  • Extreme Motion: The random augmentation used for training was capped at 30 degrees and 50 pixels. It is unclear how the model handles extreme "action-cam" style shakes that exceed these bounds.
  • Inpainting Artifacts: While Margin-Net is effective, inpainting based on adjacent frames can occasionally struggle with very fast-moving backgrounds or complex textures that haven't appeared in recent frames.
  • Hardware Integration: The paper notes the method runs on a single GPU; adapting this to the specialized NPU (Neural Processing Unit) of a smartphone would be the logical next step for consumer use.

Summary updated

Abstract

Videos are a popular media form, where online video streaming has recently gathered much popularity. In this work, we propose a novel method of real-time video stabilization - transforming a shaky video to a stabilized video as if it were stabilized via gimbals in real-time. Our framework is trainable in a self-supervised manner, which does not require data captured with special hardware setups (i.e., two cameras on a stereo rig or additional motion sensors). Our framework consists of a transformation estimator between given frames for global stability adjustments, followed by scene parallax reduction module via spatially smoothed optical flow for further stability. Then, a margin inpainting module fills in the missing margin regions created during stabilization to reduce the amount of post-cropping. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-the-art real-time video stabilization methods as well as offline methods that require camera trajectory optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps regardless of resolution (e.g., 480p or 1080p).
BibTeX
@inproceedings{choi2021self,
  author = {Jinsoo Choi and Jaesik Park and In So Kweon},
  title = {{Self-Supervised Real-time Video Stabilization}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=JYmHAW5s5FEa},
}

Loading related content...

Figure