High Quality Disparity Remapping with Two-Stage Warping

Bing Li, Chia-Wen Lin, Cheng Zheng, Shan Liu, Junsong Yuan, Bernard Ghanem, C.-c. Jay Kuo

Introduction

Stereoscopic 3D content is increasingly prevalent across various platforms, from cinema to VR. However, the optimal depth range varies significantly depending on the display size and viewer comfort. Disparity remapping is the essential process of adjusting these depth levels. The primary challenge lies in modifying depth without distorting the underlying 3D geometry or 2D shapes of objects.

Existing methods often treat images as simple 2D planes. When disparity is stretched or squashed, it can inadvertently "warp" objects—for instance, turning a flat wall into a curved one. This paper addresses these issues by balancing three conflicting goals: reaching target depth, preserving 2D shapes, and maintaining 3D structural integrity.

The 3D Structure-Preserving Constraint

The core innovation of this work is a new energy term ($E^G$) that enforces 3D consistency. The fundamental insight is that if corresponding regions in the left and right images are warped identically (consistently), the 3D structure of the object is preserved.

Conceptual illustration of consistent vs. inconsistent warping on 3D structure.

As shown in the conceptual diagram above, inconsistent warping leads to distorted 3D geometry. The authors measure "warping inconsistency" by comparing the edge vectors of warped grid meshes between the two views, mathematically penalizing any asymmetry to keep 3D structures "flat" or "straight" as intended.

Two-Stage Warping Methodology

The authors propose a two-stage approach formulated as a constrained optimization problem to handle the trade-off between stability and flexibility.

Stage I: Axis-Aligned Warping

The first stage employs axis-aligned warping, which constrains warped grids to remain rectangular. This approach is highly robust, preventing "foldovers" and ensuring a convex optimization problem. This stage focuses on bringing salient objects to their target depths while maintaining their 2D shapes and 3D structures perfectly.

Stage II: Vertex-Based Warping

While axis-aligned warping is stable, its rigidity can cause distortions in background regions. In the second stage, the algorithm switches to vertex-based warping specifically for these less important areas. Using the results from Stage I as an initial guess, background vertices are allowed to move more freely to smooth out distortions introduced by the rigid first stage.

Key Results and Performance

The method was evaluated using standard benchmarks (Flickr, Middlebury), movie content (The Monkey King 2, Monster Hunt), and synthetic frames (Elephants Dream).

Comparison of disparity remapping results between the proposed method and Lang et al.

Qualitative comparisons demonstrate that while previous state-of-the-art methods like Lang et al. often "bend" the 3D scene during remapping, this two-stage approach maintains straight lines and accurate depth. Quantitatively, the method achieved a 3D structure preservation score of 0.876, significantly outperforming VWR (0.654) and DWR (0.672).

Furthermore, a subjective study with 31 participants showed that over 71% of users preferred this method over existing techniques. The optimization is also efficient, taking approximately 0.3 seconds per frame, making it nearly 5x faster than some competing warping-based methods.

Conclusion

By combining the stability of axis-aligned warping with the flexibility of vertex-based refinement, this two-stage approach effectively solves the problem of geometric distortion in disparity remapping. The key lies in the mathematical enforcement of warping consistency between views, ensuring that the 3D world remains natural even after significant depth manipulation.

Summary updated

Abstract

A high quality disparity remapping method that preserves 2D shapes and 3D structures, and adjusts disparities of important objects in stereo image pairs is proposed. It is formulated as a constrained optimization problem, whose solution is challenging, since we need to meet multiple requirements of disparity remapping simultaneously. The one-stage optimization process either degrades the quality of important objects or introduces serious distortions in background regions. To address this challenge, we propose a two-stage warping process to solve it. In the first stage, we develop a warping model that finds the optimal warping grids for important objects to fulfill multiple requirements of disparity remapping. In the second stage, we derive another warping model to refine warping results in less important regions by eliminating serious distortions in shape, disparity and 3D structure. The superior performance of the proposed method is demonstrated by experimental results.
BibTeX
@inproceedings{li2021high,
  author = {Bing Li and Chia-Wen Lin and Cheng Zheng and Shan Liu and Junsong Yuan and Bernard Ghanem and C.-C. Jay Kuo},
  title = {{High Quality Disparity Remapping with Two-Stage Warping}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00227},
}
Figure