Sim2RealVS: A New Benchmark for Video Stabilization with a Strong Baseline

Qi Rao, Xin Yu, Shant Navasardyan, Humphrey Shi

Introduction

Video stabilization is a fundamental task in computer vision, essential for improving the quality of handheld or high-motion video. While deep learning has advanced the field, the lack of high-quality, large-scale datasets remains a significant bottleneck. Real-world data collection is plagued by the "parallax problem": it is physically impossible to capture two perfectly aligned views (one shaky, one stable) simultaneously. Even minor camera separation introduces depth-dependent shifts that degrade the quality of the ground truth.

The Sim2RealVS Benchmark

To overcome the limitations of real-world data, researchers have introduced Sim2RealVS, a large-scale synthetic-to-real benchmark. By utilizing the Grand Theft Auto V (GTA5) game engine, virtual camera trajectories can be precisely controlled. This allows for the generation of perfectly synchronized video pairs where the "shaky" and "stable" cameras share the exact same optical center, effectively eliminating parallax.

Overview of the Sim2RealVS dataset generation in GTA5, showing stable vs. unstable paths across diverse environments and weather conditions.

The dataset consists of 1,327 video pairs, a significant increase over previous benchmarks like DeepStab (60 pairs). It covers a wide array of scenarios, including urban environments, mountainous terrain, and various weather conditions such as rain and different times of day.

MTSNet: Motion-Trajectory Smoothing Network

Alongside the benchmark, the authors propose MTSNet, a novel architecture designed for robust video stabilization. MTSNet operates in three key stages:

  1. Motion Correction and Completion (MCC): This module addresses failures in optical flow estimation, particularly in textureless regions (like clear skies), by using a self-supervised approach to rectify and complete motion data.
  2. Global Trajectory Smoothing (GTS): At the core of MTSNet is the Trajectory Smoothing Transformer (TST). By treating camera motion as a sequence-to-sequence problem, the TST can analyze the entire camera path to predict a globally optimized, smooth trajectory.
  3. Frame Warping: The original frames are warped into stabilized positions using grid-based homography derived from the smoothed trajectories.

MTSNet is trained using specialized losses that emphasize both Motion Magnitude Similarity (positional accuracy) and Motion Tendency Similarity (directional consistency), the latter of which is crucial for eliminating micro-stutters.

Experimental Results

MTSNet sets a new state-of-the-art across multiple benchmarks. Key findings include:

  • Superior Generalization: Despite being trained on synthetic data, MTSNet outperforms models trained on real-world data when evaluated on the NUS and DeepStab datasets.
  • Efficiency: MTSNet achieves a processing speed of approximately 80ms per frame, making it orders of magnitude faster than high-performing iterative methods like FuSta, which can take up to 8 seconds per frame.
  • Visual Fidelity: Qualitative evaluations show that MTSNet produces fewer "jello" artifacts and maintains better structural integrity in distant background elements compared to previous leading methods.

Conclusion

The Sim2RealVS benchmark demonstrates that perfectly aligned synthetic data is a powerful alternative to noisy real-world data for video stabilization. MTSNet further proves that transformer-based architectures are exceptionally well-suited for smoothing complex camera trajectories. Future work may involve using real-world IMU data to drive virtual cameras, further closing the reality gap.

Summary updated

Abstract

Video stabilization is highly desirable when videos undergo severe jittering artifacts. The difficulty of obtaining sufficient training data obstructs the development of video stabilization. In this work, we address this issue by presenting a Sim2RealVS benchmark with more than 1,300 pairs of shaky and stable videos. Our benchmark is curated by an in-game simulator with diverse scenes and various jittering effects. Moreover, we propose a simple yet strong baseline approach, named Motion-Trajectory Smoothing Network (MTSNet), by fully exploiting our Sim2RealVS data. Our MTSNet consists of three main steps: motion estimation, global trajectory smoothing and frame warping. In motion estimation, we design a Motion Correction and Completion (MCC) module to rectify the optical flow with low confidence, such as in textureless regions, thus providing more consistent motion estimation for next steps. Benefiting from our synthetic data, we can explicitly learn a Trajectory Smoothing Transformer (TST) with ground-truth supervision to smooth global trajectories. In training TST, we propose two fully-supervised losses, i.e., a motion magnitude similarity loss and a motion tendency similarity loss. After training, our TST is able to produce smooth motion trajectories for the shaky input videos. Extensive qualitative and quantitative results demonstrate that our MTSNet achieves superior performance on both synthetic and real-world data.
BibTeX
@inproceedings{rao2023sim,
  author = {Qi Rao and Xin Yu and Shant Navasardyan and Humphrey Shi},
  title = {{Sim2RealVS: A New Benchmark for Video Stabilization with a Strong Baseline}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00537},
}
Figure