Self-Supervised Real-time Video Stabilization

Jinsoo Choi, Jaesik Park, In So Kweon

Abstract

Videos are a popular media form, where online video streaming has recently gathered much popularity. In this work, we propose a novel method of real-time video stabilization - transforming a shaky video to a stabilized video as if it were stabilized via gimbals in real-time. Our framework is trainable in a self-supervised manner, which does not require data captured with special hardware setups (i.e., two cameras on a stereo rig or additional motion sensors). Our framework consists of a transformation estimator between given frames for global stability adjustments, followed by scene parallax reduction module via spatially smoothed optical flow for further stability. Then, a margin inpainting module fills in the missing margin regions created during stabilization to reduce the amount of post-cropping. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-the-art real-time video stabilization methods as well as offline methods that require camera trajectory optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps regardless of resolution (e.g., 480p or 1080p).

Introduction

Recording high-quality video on mobile devices often involves a trade-off between stability and field-of-view. While Optical Image Stabilization (OIS) handles minor jitters, Electronic Image Stabilization (EIS) typically requires aggressive cropping—a "zoom-in tax"—to create a buffer for motion compensation. Furthermore, existing AI-based stabilization solutions often require supervised training with expensive gimbal-stabilized ground truth data or are too computationally intensive for real-time applications. This paper introduces a self-supervised, real-time framework that achieves high-quality stabilization at 41 FPS while significantly reducing the need for cropping.

Methodology

The proposed framework utilizes a three-stage cascade architecture that processes video frames sequentially to achieve both global and local stability.

Detailed architecture of the stabilization framework, showing the internal structures of Coarse-Net, Fine-Net, and Margin-Net.

1. Coarse-Net (Global Stability)

This module estimates rigid 3-DoF transformations (rotation and 2D translation) between frames. By using a U-Net architecture to process optical flow and applying a moving average filter, it smooths out abrupt camera jumps. The authors found that a 3-DoF model is more effective than higher-complexity models, which can introduce unwanted wobble artifacts.

2. Fine-Net (Detail Stability)

To address complex issues like parallax, where objects at different depths move independently, Fine-Net generates a spatially smoothed optical flow map. This allows for image warping that fixes local instabilities. Crucially, the network is designed to be foreground-agnostic; by using average pooling and down-sampling, it ignores high-frequency noise from moving objects (like pedestrians) and focuses on the low-frequency global motion of the background.

3. Margin-Net (The "Anti-Zoom" Tool)

Instead of the traditional approach of cropping the messy edges created by stabilization, Margin-Net uses information from adjacent frames to inpaint the missing boundary regions. This generative approach allows the final output to maintain a much wider field of view compared to standard EIS methods.

Self-Supervised Training

The model is trained using "Learning by Random Augmentation." By taking a single stable frame and applying synthetic "shakes" (random rotation and translation), the network learns to reverse these transformations without needing ground truth stabilized video. Temporal consistency losses are also employed to ensure smooth alignment between adjacent frames in real sequences.

Results and Performance

The entire pipeline is highly efficient, taking approximately 24.3 ms per frame (41 FPS) regardless of resolution. This makes it viable for real-time use on mobile hardware.

Visualization of the stabilization pipeline stages, showing the transition from a shaky input sequence to a smoothed, margin-inpainted result.

Quantitative evaluations show that the method outperforms state-of-the-art real-time stabilizers in three key metrics: cropping ratio, distortion value, and stability score. Qualitatively, the results exhibit significantly less blur and a wider field of view than commercial solutions like Adobe Premiere’s Warp Stabilizer or standard smartphone OIS+EIS combinations.

Conclusion

This work demonstrates that high-quality, real-time video stabilization can be achieved through a self-supervised cascade architecture. By combining coarse-to-fine stabilization with intelligent margin inpainting, the system avoids the aggressive cropping typical of modern smartphones. Future work may involve testing the model against more extreme motion profiles and optimizing the architecture for specialized mobile Neural Processing Units (NPUs).

Summary updated

Abstract

Videos are a popular media form, where online video streaming has recently gathered much popularity. In this work, we propose a novel method of real-time video stabilization - transforming a shaky video to a stabilized video as if it were stabilized via gimbals in real-time. Our framework is trainable in a self-supervised manner, which does not require data captured with special hardware setups (i.e., two cameras on a stereo rig or additional motion sensors). Our framework consists of a transformation estimator between given frames for global stability adjustments, followed by scene parallax reduction module via spatially smoothed optical flow for further stability. Then, a margin inpainting module fills in the missing margin regions created during stabilization to reduce the amount of post-cropping. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-the-art real-time video stabilization methods as well as offline methods that require camera trajectory optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps regardless of resolution (e.g., 480p or 1080p).
BibTeX
@inproceedings{choi2021self,
  author = {Jinsoo Choi and Jaesik Park and In So Kweon},
  title = {{Self-Supervised Real-time Video Stabilization}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=JYmHAW5s5FEa},
}
Figure