High-Resolution Depth Estimation for 360deg Panoramas Through Perspective and Panoramic Depth Images Registration

Chi-Han Peng, Jiayao ZhangWACV 2023

TL;DR

This paper introduces a novel, faster, and more accurate method for generating high-resolution depth maps from 360° panoramas. By registering individual perspective depth estimates to a reference panoramic depth map, they overcome the global consistency challenges of previous stitching-based approaches, leading to significantly improved detail and accuracy.

What Problem Does This Paper Address?

Estimating depth from 360° panoramic images is crucial for applications like virtual reality, 3D scene understanding, and novel view synthesis. Traditional monocular depth estimation methods designed for perspective images don't directly apply to panoramas due to their wide field of view. While specialized panoramic depth estimation methods exist, they often suffer from limitations:

  1. Resolution Constraints: Current deep learning models are often limited to outputting low-resolution depth maps (e.g., 1024x512), which is insufficient for high-fidelity VR experiences and significantly lower than the native resolution of modern 360° cameras (often >4096x2048).
  2. Lack of Detail: Panoramic methods tend to produce depth maps with less fine-grained detail compared to perspective methods, likely due to smaller and less diverse training datasets.

To address these issues, prior work proposed stitching together multiple perspective depth maps. However, this approach faces a significant challenge: aligning the scales and shifts of these individual perspective maps to achieve global consistency is computationally expensive, often involving complex disparity map alignment and Poisson blending strategies.

Methods

The proposed method, "High-Resolution Depth Estimation for 360° Panoramas Through Perspective and Panoramic Depth Images Registration," tackles the global consistency problem with a more straightforward approach:

  1. Equirectangular-to-Perspective Partitions: The input panorama is divided into several rectangular regions. For each region, a state-of-the-art perspective depth estimation method (LeReS is used) is applied to generate a perspective depth map.
  2. Reference Panoramic Depth Map: A "reference" panoramic depth map is generated in real-time using an existing panorama-based method (like SliceNet or UniFuse). This reference map is assumed to have globally consistent depth values.
  3. Perspective-to-Equirectangular Registration: Instead of complex alignment, the individual perspective depth maps are registered to the reference panoramic depth map. This is achieved by fitting a low-degree polynomial (cubic, quadratic, or linear) to a set of sampled pixels from the perspective map and their corresponding pixels in the reference map. This step efficiently aligns scales and shifts with minimal computational cost.
  4. Laplacian-based Poisson Blending: After registration, minor seams might still exist. To address this, the authors employ a Poisson-based blending approach. They optimize for a final panoramic depth map that matches the Laplacians (gradient information) of the registered perspective depth maps. A small regularization term using L2 distances to the reference panoramic depth map is added to ensure stability. This blending is performed efficiently using a multi-scale Jacobi iterative method.

Datasets

The paper evaluates its method on the following datasets:

  • Matterport3D: A real-world indoor scene dataset with ground-truth depth.
  • Replica360: A synthetic dataset of indoor scenes with ground-truth depth.
  • Stanford2D3D: Another real-world indoor scene dataset with ground-truth depths.

The authors note that their method performs well on datasets unseen during the training of the underlying perspective depth estimation models.

Key Insights and Ideas

  • Leveraging a Reference Map: The core innovation is using an existing, globally consistent panoramic depth map as a "target" for registering individual perspective depth estimates. This elegantly sidesteps the complex and computationally expensive global alignment problem.
  • Simplified Blending: By first aligning scales and shifts via registration, the subsequent Poisson blending step becomes much simpler and more effective, avoiding the need for complex spatial-varying weights.
  • Efficiency Gain: The registration-based approach is significantly faster than previous stitching methods that relied on extensive alignment and blending.

Key Results

  • Speed: The proposed method is 3.05x faster for 2K outputs and 1.94x faster for 4K outputs compared to the state-of-the-art stitching method [23].
  • Accuracy: The method achieves quantitatively better results than [23] on unseen datasets (Replica360) and competitive results on Matterport. It also improves the quantitative scores of the panorama-based methods it uses for reference map generation.
  • Qualitative: The method generates qualitatively better results with finer details than existing panorama-based methods and [23], producing clearer images with fewer inconsistencies. Figure 5 showcases these qualitative improvements.

Limitations and Future Work

  • Dependence on Reference Map: The quality of the output is inherently dependent on the quality of the reference panoramic depth map used. If the reference map contains major errors (e.g., for outdoor scenes), the final result will be negatively affected.
  • Perspective Depth Estimation Accuracy: Similar to [23], the accuracy of the input perspective depth maps also impacts the final output.
  • Future Work: The authors aim to further improve speed by developing GPU-based Laplacian solvers or exploring style transfer-based approaches for the blending step. They also acknowledge that their method can benefit from advances in both panoramic and perspective depth estimation techniques.

Summary updated

BibTeX
@inproceedings{peng2023high,
  author = {Chi-Han Peng and Jiayao Zhang},
  title = {{High-Resolution Depth Estimation for 360deg Panoramas Through Perspective and Panoramic Depth Images Registration}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure