Perceptual Real-Time 2D-to-3D Conversion Using Cue Fusion

Thomas Leimkühler, Petr Kellnhofer, Tobias Ritschel, Karol Myszkowski, Hans‐Peter Seidel

Abstract

We propose a system to infer binocular disparity from a monocular video stream in real-time. Different from classic reconstruction of physical depth in computer vision, we compute perceptually plausible disparity, that is numerically inaccurate, but results in a very similar overall depth impression with plausible overall layout, sharp edges, fine details and agreement between luminance and disparity. We use several simple monocular cues to estimate disparity maps and confidence maps of low spatial and temporal resolution in real-time. These are complemented by spatially-varying, appearance-dependent and class-specific disparity prior maps, learned from example stereo images. Scene classification selects this prior at runtime. Fusion of prior and cues is done by means of robust MAP inference on a dense spatio-temporal conditional random field with high spatial and temporal resolution. Using normal distributions allows this in constant-time, parallel per-pixel work. We compare our approach to previous 2D-to-3D conversion systems in terms of different metrics, as well as a user study and validate our notion of perceptually plausible disparity.

Introduction

Traditional 2D-to-3D conversion is a notoriously difficult problem in computer vision. Accurate reconstruction typically requires Structure-from-Motion (SfM) or complex multi-view geometry, which are computationally expensive and often struggle with non-rigid objects or real-time constraints.

This paper argues that for entertainment and general viewing, perfect physical depth is unnecessary. Human vision is remarkably forgiving of absolute depth errors as long as the depth "looks" plausible—meaning it has a consistent global layout, sharp edges that align with the image luminance, and temporal stability. The challenge is creating a system that can infer this plausible depth across diverse scenes (forests, cities, portraits) instantly and automatically.

Methodology

The core of the proposed approach is Cue Fusion, a system that combines several weak indicators of depth with learned scene-specific priors.

Overview of the system architecture, illustrating the pre-processing of scene priors and the runtime cue fusion and inference pipeline.

Monocular Cue Extraction

The system extracts six real-time cues: defocus (blur), aerial perspective (haze/color shifts), vanishing points, static occlusions (T-junctions), motion parallax, and optional user input. These cues provide initial, low-resolution estimates of disparity and confidence.

Learned Scene-Specific Priors

In a pre-processing step, the authors learned "disparity priors" for 10 different scene classes (e.g., "Forest," "Street," "Open Country"). These priors capture the typical global depth structure of a scene type. At runtime, a simple SVM classifier identifies the scene type from the 2D input to select the correct prior.

Learned disparity priors and confidence/precision maps for various scene categories. These priors provide a global layout that guides the final disparity estimation.

Research indicates that these learned priors contribute roughly 50% of the final depth information, providing the essential global layout that individual cues then refine.

MAP Inference and Fusion

Cues and priors are represented as normal distributions (mean and confidence). They are fused using a Maximum A Posteriori (MAP) estimate on a dense spatio-temporal Conditional Random Field (CRF). This fusion is performed in parallel on the GPU, allowing for constant-time processing per pixel. To ensure visual comfort, the system enforces a "disparity gradient limit," preventing distortions that could cause eye strain.

Evaluation and Results

The system was evaluated using the NYU Range Dataset and the Make3D Dataset, as well as a custom database of ~400 images across 10 classes.

Performance

The system runs at 35 FPS for 720p HD video on a mid-range GPU (GTX 780), making it suitable for real-time applications. Despite being optimized for speed and "look," the system achieved PSNR and SSIM scores comparable to heavy-duty offline reconstruction algorithms.

User Preference

In a perceptual study with 17 participants, the system was preferred over state-of-the-art offline methods (which take significantly longer to process) in 60% to 70% of cases. This validates the hypothesis that perceptual plausibility is a viable and often superior goal for 2D-to-3D conversion.

Discussion and Conclusion

A key insight of this work is that human depth perception is strongly "attracted" to luminance edges. By explicitly aligning depth discontinuities with image edges, the system avoids common artifacts like "cardboarding." Additionally, the system leverages temporal sparsity; disparity is calculated at "keyframes" (~3Hz) and propagated smoothly using motion-compensated filtering, significantly reducing computational load.

While effective, the system faces limitations in scenes with high-contrast textures (which can be mistaken for T-junctions) or extreme motion. Future work may involve integrating more sophisticated cues and exploring fusion with other sensor data. In summary, by prioritizing perceptual plausibility over physical accuracy, we achieve high-quality, real-time 2D-to-3D conversion that aligns with human visual expectations.

Summary updated

Abstract

We propose a system to infer binocular disparity from a monocular video stream in real-time. Different from classic reconstruction of physical depth in computer vision, we compute perceptually plausible disparity, that is numerically inaccurate, but results in a very similar overall depth impression with plausible overall layout, sharp edges, fine details and agreement between luminance and disparity. We use several simple monocular cues to estimate disparity maps and confidence maps of low spatial and temporal resolution in real-time. These are complemented by spatially-varying, appearance-dependent and class-specific disparity prior maps, learned from example stereo images. Scene classification selects this prior at runtime. Fusion of prior and cues is done by means of robust MAP inference on a dense spatio-temporal conditional random field with high spatial and temporal resolution. Using normal distributions allows this in constant-time, parallel per-pixel work. We compare our approach to previous 2D-to-3D conversion systems in terms of different metrics, as well as a user study and validate our notion of perceptually plausible disparity.
BibTeX
@article{leimkuhler2017perceptual,
  author = {Thomas Leimkühler and Petr Kellnhofer and Tobias Ritschel and Karol Myszkowski and Hans‐Peter Seidel},
  title = {{Perceptual Real-Time 2D-to-3D Conversion Using Cue Fusion}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2017},
  doi = {10.1109/tvcg.2017.2703612},
}
Figure