TL;DR
This paper introduces a cost-effective system for generating dense 360° depth maps by combining an omnidirectional camera with a standard, limited-FoV depth sensor. Their novel ODE-CNN architecture effectively extends the captured depth information to the entire panoramic view, significantly outperforming existing methods.
What Problem Does This Paper Address?
Autonomous robots and other applications benefit greatly from 360° cameras, which offer a wide field of view. However, obtaining dense depth information for these panoramic images is challenging and expensive. While dedicated omnidirectional depth sensors exist, they are often prohibitively costly or complex to implement (e.g., multiple sensors, spinning LiDAR). Existing methods that rely solely on monocular omnidirectional images for depth estimation can suffer from scale ambiguity and struggle with accurate structural details. This paper aims to bridge this gap by proposing a practical and affordable solution that leverages the strengths of both standard depth sensors and advanced deep learning techniques.
Methods
The core of the proposed solution is the Omnidirectional Depth Extension Convolutional Neural Network (ODE-CNN). This network is designed to take an omnidirectional image and a partial depth map (from a limited-FoV sensor) as input and output a complete, dense depth map for the entire 360° scene.
Key novel components include:
- Spherical Feature Transform Layer (SFTL): This layer is placed at the end of the encoder. It addresses the distortion inherent in equirectangular projections by re-sampling features from the omnidirectional coordinate space to a projective coordinate space. This makes feature learning more manageable. The authors further enhance this by incorporating a deformable convolution within SFTL, allowing the sampling neighborhood to be learned adaptively from the data, leading to better results.
- Deformable Convolutional Spatial Propagation Network (D-CSPN): This module is appended to the decoder. It acts as a depth refinement step, borrowing ideas from Convolutional Spatial Propagation Networks (CSPN) for depth completion. D-CSPN dynamically learns a proper context to align estimated depth structures with the reference image, significantly improving visual quality and recovering fine details. The authors also explore variations like Inverse Gnomonic CSPN (IG-CSPN) before arriving at the deformable version.
The overall architecture follows an encoder-decoder structure with mirror skip connections, building upon a ResNet-34 backbone.
Datasets
The primary dataset used for training and evaluation is the 360D dataset. This dataset is a large-scale indoor spherical RGB-D dataset that aggregates data from:
- Stanford 2D-3D
- Matterport 3D
- SunCG (computer-generated)
- SceneNet (computer-generated)
The authors use the same data split strategy as a previous state-of-the-art method (Zioulis et al., 2018) for fair comparison.
Key Insights and Ideas
- Hybrid Sensing is Key: Combining a standard, affordable depth sensor with an omnidirectional camera offers a practical and cost-effective path to dense 360° depth sensing, avoiding the need for expensive specialized hardware.
- Distortion is a Major Hurdle: Equirectangular projections inherently distort features. The SFTL's ability to transform features to a more manageable projective space is crucial for effective learning.
- Learned Deformations Unlock Detail: Simply transforming features isn't enough. Allowing the network to learn how to deform the sampling neighborhoods (via deformable convolutions in both SFTL and D-CSPN) significantly improves accuracy and structural recovery.
- Refinement is Essential: Even with good feature extraction, a dedicated refinement module like D-CSPN is necessary to recover fine geometric details and ensure smooth transitions between sensor-provided depth and estimated depth.
Key Results
The proposed ODE-CNN system achieves state-of-the-art performance on the 360D dataset.
- Significant Error Reduction: The system achieves a 33.4% relative error reduction compared to previous state-of-the-art methods on the 360D dataset.
- Improved Accuracy: The best configuration of ODE-CNN (combining DIGT and D-CSPN with partial depth input) achieves an Abs Rel of 0.0467, outperforming the previous best by a notable margin.
- Efficiency: The system is designed for efficiency, with a testing speed of 50ms per image on a single NVIDIA P40 GPU.
- Qualitative Improvements: Visual comparisons show that ODE-CNN produces depth maps with more accurate absolute scales and reveals finer scene structures compared to existing methods.
Limitations and Future Work
- Hardware Dependency: While the system is low-cost, it still relies on having both an omnidirectional camera and a separate projective depth sensor.
- Efficiency for Deployment: While 50ms per image is good, further improvements in efficiency are suggested for on-device deployment.
- Future Work: The authors express interest in further improving the efficiency of their system for real-time applications on embedded devices.
Summary updated