Cinematic-L1 Video Stabilization with a Log-Homography Model

Arwen Bradley, Jason Klivington, Joseph Triscari, Rudolph van der Merwe

Abstract

We present a method for stabilizing handheld video that simulates the camera motions cinematographers achieve with equipment like tripods, dollies, and Steadicams. We formulate a constrained convex optimization problem minimizing the $\ell_1$-norm of the first three derivatives of the stabilized motion. Our approach extends previous work by solving with full homographies in log-homography space to correct perspective while preserving linearity. Our method is computationally efficient, running at 300fps on an iPhone XS, and yields high-quality results across various challenging video sequences.

Introduction

Capturing professional-looking video with handheld devices is challenging due to inherent camera shake. While professional cinematographers use specialized hardware like tripods for static shots or dollies for smooth pans, casual users rely on software stabilization. Existing 2D stabilization methods often use affine transforms, which fail to correct perspective ("keystone") distortion, leading to visual artifacts. Conversely, 3D methods are often too slow for mobile real-time applications. We propose a 2D method that utilizes full homographies for perspective correction, optimized efficiently in a novel log-homography space.

Methodology: Sparse Motion Optimization

The core of our approach is the observation that professional camera moves are mathematically "sparse." By minimizing the $\ell_1$-norm of the motion's derivatives, we encourage the camera trajectory to "snap" into specific states.

The $\ell_1$ Heuristic

We minimize the $\ell_1$-norm of the first, second, and third derivatives of the camera's path. A zero first derivative corresponds to a static "tripod" shot; a zero second derivative represents a constant-velocity "pan"; and a zero third derivative indicates constant acceleration.

The $\ell_1$ heuristic encourages sparse derivatives, effectively simulating professional camera equipment by forcing motion into discrete states like static tripods or linear pans.

Log-Homography Transform

To handle perspective correction without the non-linearity of standard homographies, we employ Lie theory. By mapping homographies into a "log-space" vector field, complex compositions become simple additions. This allows us to formulate the stabilization as a Quadratic Program (QP), which can be solved efficiently. We further incorporate linearized constraints for cropping and field-of-view preservation using Jacobians.

Results and Evaluation

Our method was evaluated on a benchmark of 174 videos featuring challenges such as parallax, crowds, and quick rotations.

Performance and Stability

The algorithm achieves a processing speed of 300fps on an iPhone XS, making it suitable for real-time mobile use. Compared to standard smoothing techniques, our $\ell_1$ optimization provides a much more stable trajectory that respects the intended motion of the user while removing high-frequency jitter and perspective wobble.

Comparison of raw input motion, standard Gaussian smoothing, and our proposed $\ell_1$ optimization. The $\ell_1$ approach (blue) creates significantly flatter, more stable trajectories in both translation and perspective (key) parameters.

Quality Comparison

By solving for full homographies rather than affine approximations, we effectively eliminate "keystone wobble." Quantitative metrics show our method is competitive with state-of-the-art desktop solutions like Adobe Warp Stabilizer while being significantly faster.

Conclusion

Cinematic-L1 stabilization bridges the gap between fast 2D methods and high-quality 3D stabilization. By leveraging the sparsity of professional camera moves and the mathematical efficiency of log-homography space, we provide a solution that is both high-performance and visually superior for mobile video applications. Future work includes automating weight tuning through machine learning to further refine the "virtual cinematographer" experience.

Summary updated

Abstract

We present a method for stabilizing handheld video that simulates the camera motions cinematographers achieve with equipment like tripods, dollies, and Steadicams. We formulate a constrained convex optimization problem minimizing the ℓ 1 -norm of the first three derivatives of the stabilized motion. Our approach extends the work of Grundmann et al. [9] by solving with full homographies (rather than affinities) in order to correct perspective, preserving linearity by working in log-homography space. We also construct crop constraints that preserve field-of-view; model the problem as a quadratic (rather than linear) program to allow for an ℓ 2 term encouraging fidelity to the original trajectory; and add constraints and objectives to reduce distortion. Furthermore, we propose new methods for handling salient objects via both inclusion constraints and centering objectives. Finally, we describe a windowing strategy to approximate the solution in linear time and bounded memory. Our method is computationally efficient, running at 300fps on an iPhone XS, and yields high-quality results, as we demonstrate with a collection of stabilized videos, quantitative and qualitative comparisons to [9] and other methods, and an ablation study.
BibTeX
@inproceedings{bradley2021cinematic,
  author = {Arwen Bradley and Jason Klivington and Joseph Triscari and Rudolph van der Merwe},
  title = {{Cinematic-L1 Video Stabilization with a Log-Homography Model}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=0xxSBz4IsB},
}
Figure