Joint Subspace Stabilization for Stereoscopic Video

Feng Liu, Yuzhen Niu, Hailin JinICCV 2013 · OpenReview

Abstract

Shaky stereoscopic video is not only unpleasant to watch but often causes significant 3D fatigue. Standard monocular stabilization methods, when applied to left and right views independently, tend to introduce vertical disparities and damage horizontal depth cues, destroying the stereoscopic experience. This paper presents a joint subspace stabilization method. We prove that feature trajectories from both views share the same low-rank subspace. By jointly computing and smoothing this common subspace, we stabilize both videos simultaneously, satisfying stereoscopic constraints without the need for expensive 3D reconstruction or explicit left-right correspondence.

Introduction

Stabilizing 3D video presents unique challenges compared to 2D video. Independent 2D stabilization often results in "vertical disparity"—where corresponding points in the left and right eyes appear at different heights. This physical impossibility leads to immediate eye strain. Furthermore, horizontal disparity, the primary cue for depth, can be distorted, leading to a warped or jittery sense of space.

Existing solutions typically fall into two categories: robust but constraint-ignorant 2D methods, or theoretically sound but "brittle" 3D reconstruction (Structure-from-Motion) techniques. The proposed method offers a middle ground: the robustness of 2D subspace methods combined with the geometric integrity required for stereo.

Joint Subspace Methodology

The core innovation is the Joint Subspace Stabilization algorithm. It is built upon the mathematical proof that for a stereoscopic camera rig, the feature trajectories from both cameras lie within the same low-rank subspace (typically rank 9).

Subspace approximation error decreases as more trajectories (m) are incorporated, demonstrating the stability of the joint approach.

The process involves four key steps:

  1. Feature Tracking: Using KLT trackers to identify features in both views independently.
  2. Matrix Factorization: Assembling these into a "joint trajectory matrix" and decomposing it into scene structure (coefficients) and camera motion (eigen-trajectories).
  3. Joint Smoothing: Smoothing the shared eigen-trajectories, which ensures both views remain synchronized.
  4. Warping: Applying content-preserving warping to generate the final stabilized frames.

Crucially, this approach does not require explicit left-right point matching, a common failure point in other stereo algorithms.

Experimental Results

The method was tested on 57 stereoscopic videos of varying lengths and content. The joint method successfully stabilized 55 videos, significantly outperforming independent stabilization, which failed on 9.

Comparison of stabilization results: original shaky video (left), separate stabilization (middle), and the proposed joint stabilization (right). The joint method preserves stereo alignment.

Quantitatively, the joint method maintained an average vertical disparity of 0.68 pixels, effectively preserving the original alignment. In contrast, separate stabilization increased vertical disparity peaks to over 1.6 pixels in many cases. Qualitatively, the results avoid the "eye-pulling" sensations associated with poor 3D stabilization. The system operates at approximately 4 fps on consumer hardware, making it much more efficient than full 3D reconstruction.

Conclusion

By leveraging the shared subspace of stereoscopic trajectories, this method provides a robust and efficient solution for 3D video stabilization. It avoids the pitfalls of independent 2D methods and the complexity of 3D reconstruction, ensuring a comfortable and high-quality viewing experience for stereoscopic content.

Summary updated

Abstract

Shaky stereoscopic video is not only unpleasant to watch but may also cause 3D fatigue. Stabilizing the left and right view of a stereoscopic video separately using a monocular stabilization method tends to both introduce undesirable vertical disparities and damage horizontal disparities, which may destroy the stereoscopic viewing experience. In this paper, we present a joint subspace stabilization method for stereoscopic video. We prove that the low-rank subspace constraint for monocular video [10] also holds for stereoscopic video. Particularly, the feature trajectories from the left and right video share the same subspace. Based on this proof, we develop a stereo subspace stabilization method that jointly computes a common subspace from the left and right video and uses it to stabilize the two videos simultaneously. Our method meets the stereoscopic constraints without 3D reconstruction or explicit left-right correspondence. We test our method on a variety of stereoscopic videos with different scene content and camera motion. The experiments show that our method achieves high-quality stabilization for stereoscopic video in a robust and efficient way.
BibTeX
@inproceedings{liu2013joint,
  author = {Feng Liu and Yuzhen Niu and Hailin Jin},
  title = {{Joint Subspace Stabilization for Stereoscopic Video}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=ByW6JbGOZH},
}
Figure