360° Depth Estimation from Multiple Fisheye Images with Origami Crown Representation of Icosahedron

Ren Komatsu, Hiromitsu Fujii, Yusuke Tamura, Atsushi Yamashita, Hajime Asama

Introduction

360° depth perception is a cornerstone for autonomous navigation in indoor environments. While LiDAR provides high accuracy, its cost and mechanical complexity often favor camera-based solutions. Fisheye cameras, in particular, offer a wide field of view (FOV) but introduce significant radial distortion that challenges standard Convolutional Neural Networks (CNNs). This paper presents a novel approach to 360° depth estimation using a plane-sweeping stereo pipeline based on an icosahedron representation, termed "CrownConv."

Methodology

The core of the proposed method is the decoupling of the camera hardware from the neural network weights by projecting raw fisheye images onto a standardized icosahedron grid.

Icosahedron-based Representation

Instead of using equirectangular projections which suffer from severe distortion at the poles, the authors utilize an icosahedron—a 20-sided polyhedron that provides a more uniform sampling of the sphere. To process this data efficiently with standard 2D CNNs, the icosahedron is unfolded into a specific layout resembling an "origami crown." This "CrownConv" representation allows the use of optimized 2D convolutions while maintaining the global context of the spherical environment.

Different image representations used in the pipeline: (a) raw fisheye, (b) equirectangular, (c) the proposed icosahedron-based 'origami crown' unfolding, and (d) the estimated depth map.

Icospherical Sweeping and Depth Regression

The pipeline employs a plane-sweeping stereo approach adapted for spherical geometry. Virtual spheres are swept at varying radii to construct a cost volume directly on the icosahedron grid. This cost volume is regularized using a 3D version of CrownConv, and the final continuous depth map is extracted via a soft argmax function. By aligning the icosahedron with the gravity vector using extrinsic camera parameters, the system remains robust to variations in physical camera mounting.

Experiments and Results

The model was evaluated on synthetic datasets (OmniThings and OmniHouse) and a newly introduced "Rotated OmniHouse" dataset designed to test robustness against camera misalignment.

Robustness to Alignment

A key finding is the system's resilience to changes in the camera rig. While traditional models like OmniMVS saw their Root Mean Square (RMS) error more than double when cameras were rotated by 45°, the proposed IcoSweepNet maintained stable performance (RMS error of 2.53 vs 5.17 for the baseline).

Computational Efficiency

The method is highly efficient, processing four fisheye images into a 360° depth map in approximately 0.42 to 0.73 seconds on a laptop GPU (NVIDIA RTX 2080 Max-Q). This speed, combined with its robustness, makes it suitable for real-time robotics applications.

Conclusion

The CrownConv architecture and icosahedron-based sweeping provide a robust and efficient framework for omnidirectional depth estimation. By standardizing the input space and leveraging efficient 2D convolutions, the method overcomes the limitations of traditional fisheye processing. Future work will focus on bridging the sim-to-real gap regarding reflective surfaces and further optimizing inference speed.

Summary updated

Abstract

In this study, we present a method for all-around depth estimation from multiple omnidirectional images for indoor environments. In particular, we focus on plane-sweeping stereo as the method for depth estimation from the images. We propose a new icosahedron-based representation and ConvNets for omnidirectional images, which we name "CrownConv" because the representation resembles a crown made of origami. CrownConv can be applied to both fisheye images and equirectangular images to extract features. Furthermore, we propose icosahedron-based spherical sweeping for generating the cost volume on an icosahedron from the extracted features. The cost volume is regularized using the three-dimensional CrownConv, and the final depth is obtained by depth regression from the cost volume. Our proposed method is robust to camera alignments by using the extrinsic camera parameters; therefore, it can achieve precise depth estimation even when the camera alignment differs from that in the training dataset. We evaluate the proposed model on synthetic datasets and demonstrate its effectiveness. As our proposed method is computationally efficient, the depth is estimated from four fisheye images in less than a second using a laptop with a GPU. Therefore, it is suitable for real-world robotics applications. Our source code is available at https://github.com/matsuren/crownconv360depth.
BibTeX
@inproceedings{komatsu2020depth,
  author = {Ren Komatsu and Hiromitsu Fujii and Yusuke Tamura and Atsushi Yamashita and Hajime Asama},
  title = {{360° Depth Estimation from Multiple Fisheye Images with Origami Crown Representation of Icosahedron}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Xiyq6mu7CZ},
}
Figure