Abstract
In this paper, we propose a novel end-to-end deep neural network model for omnidirectional depth estimation from a wide-baseline multi-view stereo setup. The images captured with ultra wide field-of-view (FOV) cameras on an omnidirectional rig are processed by the feature extraction module, and then the deep feature maps are warped onto the concentric spheres swept through all candidate depths using the calibrated camera parameters. The 3D encoder-decoder block takes the aligned feature volume to produce the omnidirectional depth estimate with regularization on uncertain regions utilizing the global context information. In addition, we present large-scale synthetic datasets for training and testing omnidirectional multi-view stereo algorithms. Our datasets consist of 11K ground-truth depth maps and 45K fisheye images in four orthogonal directions with various objects and environments. Experimental results show that the proposed method generates excellent results in both synthetic and real-world environments, and it outperforms the prior art and the omnidirectional versions of the state-of-the-art conventional stereo algorithms.
Introduction
Standard depth estimation usually relies on a pair of forward-facing cameras with a limited field-of-view (FOV). For autonomous robots and self-driving cars, seeing only what’s in front isn't enough—they need a "human-like" or holonomic awareness of their entire surroundings. While one could stitch several standard stereo pairs together, this creates two major problems: the "Seam Problem," where discontinuities appear at camera boundaries, and a "Loss of Context," where algorithms miss global geometric cues. Furthermore, wide-baseline omnidirectional setups suffer from "multiple true matches," where a single ray might appear to hit different objects when viewed from different angles, confusing traditional algorithms.
Methods
The authors propose OmniMVS, a pipeline consisting of three main stages:
- Feature Extraction: A 2D CNN (sharing weights across all cameras) extracts deep feature maps from four 220° FOV fisheye images. Warping feature maps rather than raw pixels is computationally efficient and allows the network to learn how to handle fisheye distortion within the feature extraction phase.
- Spherical Sweeping: Instead of the "plane sweeping" used in conventional stereo, the model warps these features onto a series of concentric spheres at various depth hypotheses. This aligns the features in a 3D space that represents the world around the rig.
- Cost Volume & 3D Regularization: The aligned features are concatenated into a 4D volume. A 3D encoder-decoder processes this volume, allowing the network to learn global context and smooth out uncertain regions (like reflections or textureless walls) before a final depth is calculated via a softargmin layer.
Datasets
Because existing stereo datasets (like KITTI or SceneFlow) use narrow-FOV pinhole models, the authors created two massive new synthetic benchmarks:
- OmniThings: 11,000+ scenes featuring random 3D objects from ShapeNet scattered around the rig in various environments.
- OmniHouse: Realistic indoor environments based on the SUNCG dataset, providing 2,500+ frames of complex room layouts.
They also evaluated on the Sunny/Cloudy/Sunset outdoor datasets to prove the model's versatility, demonstrating that a model trained entirely on synthetic data generalizes remarkably well to real-world footage.
Key Results
OmniMVS significantly outperformed previous state-of-the-art methods like SweepNet. In the "OmniHouse" dataset, OmniMVS achieved a Mean Absolute Error (MAE) of 1.04, compared to SweepNet’s 2.07. The model also produced much cleaner results than "stitched" versions of top-tier stereo models (PSMNet and DispNet-CSS), especially at the boundaries where cameras overlap. Despite the complexity of 360° processing, the model runs in approximately 1.06 seconds on an NVIDIA 1080Ti.
Conclusion and Future Work
OmniMVS demonstrates that warping feature maps onto concentric spheres and using 3D regularization effectively handles the challenges of wide-baseline omnidirectional stereo. Future work includes addressing polar distortion at the very top and bottom of the sphere, optimizing memory for higher resolutions, and adapting to dynamic rigs where camera calibration might shift due to vibration.
Summary updated