Abstract
We propose a novel robust video stabilization method. Unlike traditional video stabilization techniques that involve complex motion models, we directly model the appearance change of the frames as the dense optical flow field of consecutive frames. We introduce a new formulation of the video stabilization task based on first principles, which leads to a large scale non-convex problem. This problem is hard to solve, so previous optical flow based approaches have resorted to heuristics. In this paper, we propose a novel optimization routine that transfers this problem into the convolutional neural network parameter domain. While we exploit the general benefits of CNNs, including standard gradient-based optimization techniques, our method is a new approach to using CNNs purely as an optimizer rather than learning from data. Our method trains the CNN from scratch on each specific input example, and intentionally overfits the CNN parameters to produce the best result on the input example. By solving the problem in the CNN weight space rather than directly for image pixels, we make it a viable formulation for video stabilization. Our method produces both visually and quantitatively better results than previous work, and is robust in situations acknowledged as limitations in current state-of-the-art methods.
Introduction
Video stabilization is the process of removing shaky camera motion while preserving the actual scene content. Traditionally, this is achieved through two primary methods: 2D methods, which track a few points and apply global transforms (like pan or zoom), and 3D methods, which attempt to reconstruct the scene's geometry. While 2D methods are fast, they often fail in the presence of parallax (where objects at different depths move differently). Conversely, 3D methods are accurate but brittle; they fail if the software cannot find enough reliable tracking points.
The ideal solution involves using dense optical flow to track every pixel. However, for a standard 480p video, this requires solving for over 40 million motion vectors. This "curse of dimensionality" creates a complex optimization landscape where standard algorithms frequently get stuck in local minima, resulting in jittery or distorted output.
Methodology: CNN as an Optimizer
This paper introduces a "test-time optimization" approach. Instead of training a network on millions of videos to learn general stabilization, the authors train a network from scratch on one specific video.
Optimization in Weight Space
Instead of directly searching for the best coordinates for every pixel (the "pixel space"), the method optimizes the weights of a CNN. The CNN’s architecture acts as a natural "prior" that encourages spatial smoothness and consistency. Inspired by concepts like "Deep Image Prior," the authors leverage the fact that CNNs are naturally biased toward producing natural-looking, spatially smooth images, which helps filter out errors in raw optical flow data.
Hybrid Warping and Regularization
The network outputs two components for every frame: a global 2D affine transformation to handle camera shake and a dense, per-pixel warp field to handle complex local motions like parallax. To maintain temporal consistency, the video is processed in overlapping 60-frame windows. Furthermore, a "linear warping constraint" is included to ensure the resulting video remains locally continuous, preventing "wobbly jelly" distortions.
Key Results and Robustness
The authors evaluated their method on 25 challenging videos and the NUS dataset benchmark. The method significantly outperformed previous state-of-the-art techniques, such as Steadyflow, in frequency domain stability (Metric D) and motion smoothness (Metric E).
A major advantage of this approach is its robustness in scenarios that typically break traditional feature-tracking methods, such as large foreground occlusions or heavy motion blur. Because it utilizes a per-pixel warp field, it can stabilize scenes with complex depth layers far more effectively than standard 2D homography-based tools.
Limitations and Future Work
The primary hurdle for this method is computation time. Because the network must be trained for every new video, it takes approximately 30 minutes to stabilize a 10-second video on a high-end GPU. This makes it an offline process unsuitable for real-time mobile use. Future directions include using a "universal" pre-trained network as a starting point, which could potentially reduce processing time through faster fine-tuning on specific input videos.
Summary updated