Omnidirectional Depth Extension Networks

Xinjing Cheng, Peng Wang, Yanqi Zhou, Chenye Guan, Ruigang Yang

TL;DR

This paper introduces a cost-effective system for generating dense 360° depth maps by combining an omnidirectional camera with a standard, limited-FoV depth sensor. Their novel ODE-CNN architecture effectively extends the captured depth information to the entire panoramic view, significantly outperforming existing methods.

What Problem Does This Paper Address?

Autonomous robots and other applications benefit greatly from 360° cameras, which offer a wide field of view. However, obtaining dense depth information for these panoramic images is challenging and expensive. While dedicated omnidirectional depth sensors exist, they are often prohibitively costly or complex to implement (e.g., multiple sensors, spinning LiDAR). Existing methods that rely solely on monocular omnidirectional images for depth estimation can suffer from scale ambiguity and struggle with accurate structural details. This paper aims to bridge this gap by proposing a practical and affordable solution that leverages the strengths of both standard depth sensors and advanced deep learning techniques.

Methods

The core of the proposed solution is the Omnidirectional Depth Extension Convolutional Neural Network (ODE-CNN). This network is designed to take an omnidirectional image and a partial depth map (from a limited-FoV sensor) as input and output a complete, dense depth map for the entire 360° scene.

Key novel components include:

  • Spherical Feature Transform Layer (SFTL): This layer is placed at the end of the encoder. It addresses the distortion inherent in equirectangular projections by re-sampling features from the omnidirectional coordinate space to a projective coordinate space. This makes feature learning more manageable. The authors further enhance this by incorporating a deformable convolution within SFTL, allowing the sampling neighborhood to be learned adaptively from the data, leading to better results.
  • Deformable Convolutional Spatial Propagation Network (D-CSPN): This module is appended to the decoder. It acts as a depth refinement step, borrowing ideas from Convolutional Spatial Propagation Networks (CSPN) for depth completion. D-CSPN dynamically learns a proper context to align estimated depth structures with the reference image, significantly improving visual quality and recovering fine details. The authors also explore variations like Inverse Gnomonic CSPN (IG-CSPN) before arriving at the deformable version.

The overall architecture follows an encoder-decoder structure with mirror skip connections, building upon a ResNet-34 backbone.

Datasets

The primary dataset used for training and evaluation is the 360D dataset. This dataset is a large-scale indoor spherical RGB-D dataset that aggregates data from:

  • Stanford 2D-3D
  • Matterport 3D
  • SunCG (computer-generated)
  • SceneNet (computer-generated)

The authors use the same data split strategy as a previous state-of-the-art method (Zioulis et al., 2018) for fair comparison.

Key Insights and Ideas

  • Hybrid Sensing is Key: Combining a standard, affordable depth sensor with an omnidirectional camera offers a practical and cost-effective path to dense 360° depth sensing, avoiding the need for expensive specialized hardware.
  • Distortion is a Major Hurdle: Equirectangular projections inherently distort features. The SFTL's ability to transform features to a more manageable projective space is crucial for effective learning.
  • Learned Deformations Unlock Detail: Simply transforming features isn't enough. Allowing the network to learn how to deform the sampling neighborhoods (via deformable convolutions in both SFTL and D-CSPN) significantly improves accuracy and structural recovery.
  • Refinement is Essential: Even with good feature extraction, a dedicated refinement module like D-CSPN is necessary to recover fine geometric details and ensure smooth transitions between sensor-provided depth and estimated depth.

Key Results

The proposed ODE-CNN system achieves state-of-the-art performance on the 360D dataset.

  • Significant Error Reduction: The system achieves a 33.4% relative error reduction compared to previous state-of-the-art methods on the 360D dataset.
  • Improved Accuracy: The best configuration of ODE-CNN (combining DIGT and D-CSPN with partial depth input) achieves an Abs Rel of 0.0467, outperforming the previous best by a notable margin.
  • Efficiency: The system is designed for efficiency, with a testing speed of 50ms per image on a single NVIDIA P40 GPU.
  • Qualitative Improvements: Visual comparisons show that ODE-CNN produces depth maps with more accurate absolute scales and reveals finer scene structures compared to existing methods.

Limitations and Future Work

  • Hardware Dependency: While the system is low-cost, it still relies on having both an omnidirectional camera and a separate projective depth sensor.
  • Efficiency for Deployment: While 50ms per image is good, further improvements in efficiency are suggested for on-device deployment.
  • Future Work: The authors express interest in further improving the efficiency of their system for real-time applications on embedded devices.

Summary updated

Abstract

Omnidirectional 360° camera proliferates rapidly for autonomous robots since it significantly enhances the perception ability by widening the field of view (FoV). However, corresponding 360° depth sensors, which are also critical for the perception system, are still difficult or expensive to have. In this paper, we propose a low-cost 3D sensing system that combines an omnidirectional camera with a calibrated projective depth camera, where the depth from the limited FoV can be automatically extended to the rest of recorded omnidirectional image. To accurately recover the missing depths, we design an omnidirectional depth extension convolutional neural network (ODE-CNN), in which a spherical feature transform layer (SFTL) is embedded at the end of feature encoding layers, and a deformable convolutional spatial propagation network (D-CSPN) is appended at the end of feature decoding layers. The former re-samples the neighborhood of each pixel in the omnidirectional coordination to the projective coordination, which reduce the difficulty of feature learning, and the later automatically finds a proper context to well align the structures in the estimated depths via CNN w.r.t. the reference image, which significantly improves the visual quality. Finally, we demonstrate the effectiveness of proposed ODE-CNN over the popular 360D dataset, and show that ODE-CNN significantly outperforms (relatively 33% reduction in depth error) other state-of-the-art (SoTA) methods.
BibTeX
@inproceedings{cheng2020omnidirectional,
  author = {Xinjing Cheng and Peng Wang and Yanqi Zhou and Chenye Guan and Ruigang Yang},
  title = {{Omnidirectional Depth Extension Networks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=8hNcAMWmwJp},
}
Figure