ODE-CNN: Omnidirectional Depth Extension Networks

Xinjing Cheng, Peng Wang, Yanqi Zhou, Chenye Guan, Ruigang Yang

Abstract

Omnidirectional 360° cameras are increasingly popular for autonomous robots as they widen the field of view (FoV) and enhance perception. However, 360° depth sensors remain expensive or difficult to implement. This paper proposes a low-cost 3D sensing system combining an omnidirectional camera with a calibrated projective depth camera. We introduce the Omnidirectional Depth Extension Convolutional Neural Network (ODE-CNN) to automatically extend depth from a limited FoV to the entire omnidirectional image. ODE-CNN incorporates a Spherical Feature Transform Layer (SFTL) to handle spherical coordination and a Deformable Convolutional Spatial Propagation Network (D-CSPN) to align estimated depths with reference images. Our method achieves a 33% reduction in depth error compared to state-of-the-art methods on the 360D dataset.

Introduction

For autonomous robots, comprehensive 360° spatial awareness is critical for navigation and obstacle avoidance. Traditional solutions like high-end LiDAR are prohibitively expensive, while multi-camera stereo setups suffer from complex calibration and high bandwidth requirements. Monocular 360° depth estimation offers a cheaper alternative but often fails to resolve metric scale ambiguity, leading to "scale confusion" where the geometry is correct but the absolute size is wrong.

Comparison of sensing setups and qualitative results showing ODE-CNN's performance against RectNet and Ground Truth.

The proposed system uses a "hybrid" approach: a single 360° camera paired with a narrow-FoV depth sensor (e.g., a Kinect). This "partial depth" (PD) serves as a metric seed that the ODE-CNN uses to hallucinate accurate, scale-correct depth for the rest of the environment.

Methodology

The ODE-CNN architecture is an encoder-decoder network specifically optimized for the unique geometry of equirectangular projections.

The ODE-CNN architecture, illustrating the flow from input RGB and partial depth to the final refined omnidirectional depth map via SFTL and D-CSPN modules.

Spherical Feature Transform Layer (SFTL)

Standard convolutions assume a flat grid, but 360° images are significantly distorted at the poles. SFTL addresses this by resampling pixel neighborhoods based on spherical geometry (inverse gnomonic projection). This allows the network to learn features as if the image were not distorted, reducing the learning burden on the model.

Deformable Convolutional Spatial Propagation Network (D-CSPN)

To sharpen depth boundaries and ensure they align with visual edges, the network employs D-CSPN at the end of the decoding stage. Unlike standard CSPN which uses a fixed local neighborhood, D-CSPN learns offsets to find the most relevant context for depth refinement. This is particularly effective for the warped perspectives found in panoramic images, resulting in significantly sharper object boundaries.

Results

The model was rigorously tested on the 360D dataset, which aggregates data from Matterport3D, Stanford 2D-3D, SunCG, and SceneNet.

  • Superior Accuracy: ODE-CNN achieved a 33% relative reduction in depth error compared to the previous state-of-the-art, RectNet.
  • Metric Precision: The system reached a relative depth error of approximately 4%, providing sufficient precision for real-world robotic tasks.
  • Efficiency: With a latency of ~50ms per frame on standard GPU hardware, the system is suitable for real-time robotic applications.
  • Visual Quality: Qualitatively, the depth maps produced by ODE-CNN show significantly better structural detail and sharper edges than previous "blobby" estimation methods.

Conclusion

ODE-CNN demonstrates that high-quality omnidirectional depth can be achieved using a low-cost, hybrid sensor setup. By leveraging geometric priors through SFTL and adaptive refinement via D-CSPN, the network effectively bridges the gap between limited-FoV depth sensing and full 360° perception. Future work will explore performance in complex outdoor settings and further optimization for deployment on edge devices.

Summary updated

Abstract

Omnidirectional 360{\deg} camera proliferates rapidly for autonomous robots since it significantly enhances the perception ability by widening the field of view(FoV). However, corresponding 360{\deg} depth sensors, which are also critical for the perception system, are still difficult or expensive to have. In this paper, we propose a low-cost 3D sensing system that combines an omnidirectional camera with a calibrated projective depth camera, where the depth from the limited FoV can be automatically extended to the rest of the recorded omnidirectional image. To accurately recover the missing depths, we design an omnidirectional depth extension convolutional neural network(ODE-CNN), in which a spherical feature transform layer(SFTL) is embedded at the end of feature encoding layers, and a deformable convolutional spatial propagation network(D-CSPN) is appended at the end of feature decoding layers. The former resamples the neighborhood of each pixel in the omnidirectional coordination to the projective coordination, which reduces the difficulty of feature learning, and the later automatically finds a proper context to well align the structures in the estimated depths via CNN w.r.t. the reference image, which significantly improves the visual quality. Finally, we demonstrate the effectiveness of proposed ODE-CNN over the popular 360D dataset and show that ODE-CNN significantly outperforms (relatively 33% reduction in-depth error) other state-of-the-art (SoTA) methods.
BibTeX
@inproceedings{cheng2020ode,
  author = {Xinjing Cheng and Peng Wang and Yanqi Zhou and Chenye Guan and Ruigang Yang},
  title = {{ODE-CNN: Omnidirectional Depth Extension Networks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=8hNcAMWmwJp},
}
Figure