360° Depth Estimation from Multiple Fisheye Images with Origami Crown Representation of Icosahedron

Ren Komatsu, Hiromitsu Fujii, Yusuke Tamura, Atsushi Yamashita, Hajime Asama

TL;DR

This paper introduces a novel method for 360° depth estimation from multiple fisheye cameras using an icosahedron-based representation called "CrownConv." By unfolding the icosahedron into an "origami crown" shape, the authors can apply highly efficient 2D CNNs to heavily distorted fisheye images. The resulting pipeline is remarkably robust to changes in camera alignment and fast enough to run in under a second on a laptop GPU, making it highly practical for real-world robotics.

What Problem Does This Paper Address?

For autonomous robots and vehicles, 360° depth perception is critical for navigation and obstacle avoidance. While LiDAR is the gold standard, it is expensive. Fisheye cameras offer a cheap, lightweight alternative with a massive Field of View (FOV).

However, applying standard Convolutional Neural Networks (CNNs) to fisheye images is notoriously difficult. Standard CNNs assume a flat, perspective image; fisheye images are heavily distorted, meaning an object's shape changes drastically depending on where it appears in the frame (breaking translation equivariance). Previous workarounds—like projecting images onto 3D spheres or using cubemaps—are either computationally massive, suffer from edge discontinuities, or completely break down if the physical camera alignment on the robot changes after the model is trained.

Methods

The authors propose a learning-based plane-sweeping stereo pipeline that operates entirely on an icosahedron (a 20-sided polyhedron that approximates a sphere) rather than on flat 2D images.

  1. Projection: Fisheye images are projected onto the icosahedron using the cameras' intrinsic and extrinsic parameters. Crucially, they align the projection with gravity (e.g., ceilings map to the north pole, floors to the south), standardizing the environment regardless of how the cameras are mounted.
  2. CrownConv: To extract features, they introduce a novel CNN architecture. They unfold the icosahedron grid and pack it into five rectangular feature maps that resemble an origami crown. This clever reshaping allows them to use standard, highly optimized 2D convolutions on spherical data without losing global context.
  3. Icospherical Sweeping: Instead of sweeping flat virtual planes to find stereo matches, they sweep virtual spheres at different radii to build a cost volume directly on the icosahedron.
  4. Depth Regression: A 3D version of CrownConv regularizes this cost volume, and a soft argmax function extracts the final continuous depth map.

Datasets

  • OmniThings & OmniHouse: Existing synthetic datasets containing 4-camera fisheye setups with ground truth depth, used for training and baseline evaluation.
  • Rotated OmniHouse (Newly Introduced): A modified version of OmniHouse created by the authors. The camera facing directions are rotated (by 15°, 30°, and 45°) to test how well models handle physical changes to the camera rig.
  • Real-World Data: Qualitative tests were run using a custom Unmanned Ground Vehicle (UGV) equipped with four downward-facing fisheye cameras.

Key Insights and Ideas

  • Gravity as an Anchor: By using extrinsic camera parameters to project images onto the icosahedron relative to gravity, the network doesn't overfit to a specific camera rig setup. You can physically bump or rotate a camera on the robot, and the model will still work without needing to be retrained.
  • The "Origami Crown" Hack: Processing spherical data usually requires custom, slow graph convolutions or creates nasty edge artifacts. The origami crown layout cleverly maps spherical geometry into standard 2D tensors. This unlocks the blazing speed of standard GPU 2D convolutions while maintaining the mathematical integrity of the 360° space.
  • Decoupling Hardware from the Model: Because the neural network operates strictly on the standardized icosahedron rather than the raw camera feeds, the hardware setup is entirely decoupled from the learned weights.

Key Results

  • Robustness to Camera Alignment: On the new "Rotated OmniHouse" dataset, the baseline model (OmniMVS) degraded severely as rotation increased. At a 45° rotation, OmniMVS's Root Mean Square (RMS) error jumped to 5.17. The proposed method (IcoSweepNet) remained highly stable, maintaining an RMS error of just 2.53.
  • Computational Efficiency: The pipeline is fast. It can estimate a full 360° depth map from four fisheye images in just 0.42 seconds (using 16 virtual spheres) or 0.73 seconds (using 32 virtual spheres) on a standard laptop GPU (NVIDIA RTX 2080 Max-Q).
  • Real-World Viability: Despite being trained entirely on synthetic data, the model successfully estimated the depth of surrounding objects when deployed on a real-world UGV.

Limitations and Future Work

The authors note a few limitations. First, in real-world testing, the model struggled with floor reflections—a classic "sim-to-real" gap, as the synthetic training data did not contain glossy floors. Second, regions with poor overlapping fields of view between the cameras (such as the very top of the sphere) yielded high depth errors.

For future work, the authors suggest training the model on real-world datasets to close the sim-to-real gap. They also plan to further optimize the codebase to achieve true real-time inference, which is necessary for high-agility robotics.

Summary updated

Abstract

In this study, we present a method for all-around depth estimation from multiple omnidirectional images for indoor environments. In particular, we focus on plane-sweeping stereo as the method for depth estimation from the images. We propose a new icosahedron-based representation and ConvNets for omnidirectional images, which we name "CrownConv" because the representation resembles a crown made of origami. CrownConv can be applied to both fisheye images and equirectangular images to extract features. Furthermore, we propose icosahedron-based spherical sweeping for generating the cost volume on an icosahedron from the extracted features. The cost volume is regularized using the three-dimensional CrownConv, and the final depth is obtained by depth regression from the cost volume. Our proposed method is robust to camera alignments by using the extrinsic camera parameters; therefore, it can achieve precise depth estimation even when the camera alignment differs from that in the training dataset. We evaluate the proposed model on synthetic datasets and demonstrate its effectiveness. As our proposed method is computationally efficient, the depth is estimated from four fisheye images in less than a second using a laptop with a GPU. Therefore, it is suitable for real-world robotics applications. Our source code is available at https://github.com/matsuren/crownconv360depth.
BibTeX
@inproceedings{komatsu2020depth,
  author = {Ren Komatsu and Hiromitsu Fujii and Yusuke Tamura and Atsushi Yamashita and Hajime Asama},
  title = {{360° Depth Estimation from Multiple Fisheye Images with Origami Crown Representation of Icosahedron}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Xiyq6mu7CZ},
}

Loading related content...

Figure