TL;DR
This paper introduces SweepNet, a novel wide-baseline omnidirectional stereo system that uses four fisheye cameras to generate dense 360° depth maps. By employing a deep neural network for cost computation on spherical images and a sphere-sweeping approach, SweepNet achieves highly accurate depth estimation in challenging scenarios, outperforming existing methods.
What Problem Does This Paper Address?
Traditional stereo vision systems, while effective for depth estimation, typically rely on two cameras facing the same direction. This creates blind spots, limiting their utility in applications requiring a comprehensive understanding of the surrounding environment, such as robotics and autonomous driving. While omnidirectional systems exist, they often use multiple cameras, complex mirror setups, or suffer from low resolution and unstable depth estimates, especially in wide-baseline configurations. Wide-baseline stereo, crucial for long-range sensing, exacerbates challenges like appearance changes and geometric distortions, making accurate depth estimation difficult. This paper tackles the problem of achieving accurate, dense, and full 360° depth estimation with a simplified wide-baseline omnidirectional system.
Methods
The core of SweepNet lies in its innovative approach to omnidirectional depth estimation. Instead of traditional methods that stitch together multiple rectified stereo pairs, SweepNet directly generates a single, dense omnidirectional depth map.
- Hardware Setup: The system utilizes four ultra-wide field-of-view (220°) fisheye cameras mounted on a wide-baseline rig, positioned to cover all cardinal directions.
- Calibration: A robust extrinsic calibration algorithm based on bundle adjustment is employed to accurately determine the relative poses of the cameras.
- Spherical Warping: Input fisheye images are warped onto concentric virtual spheres centered at the rig's coordinate system. This spherical projection normalizes the images, creating a uniform input for the subsequent neural network.
- SweepNet Architecture: A deep convolutional neural network (SweepNet) is proposed to compute the matching cost volume. Unlike traditional patch-based methods, SweepNet leverages global context within the spherical images to learn more robust correspondences, especially in wide-baseline scenarios where local information can be unreliable.
- Cost Aggregation and Refinement: The cost volume is aggregated, and then Semi-Global Matching (SGM) is used for cost aggregation and refinement.
- Depth Estimation: The final inverse depth map is obtained using a winner-takes-all strategy on the refined cost volume.
Datasets
- Synthetic Urban Datasets: Realistic synthetic urban datasets were rendered using Blender for training and testing. These datasets simulate various weather conditions (Sunny, Cloudy, Sunset) to evaluate the system's robustness.
- Real-World Datasets: The authors also collected real-world indoor and outdoor datasets to demonstrate the system's performance on actual captured data.
Key Insights and Ideas
- Global Context for Wide-Baseline Stereo: The paper highlights that traditional local patch-based matching methods struggle with the significant appearance changes and geometric distortions inherent in wide-baseline stereo. SweepNet's use of global context within spherical images is crucial for overcoming this limitation.
- Spherical Sweeping as a Unified Framework: Warping images onto virtual spheres provides a unified representation that simplifies the stereo matching process and allows for seamless handling of different camera configurations and baselines.
- Fisheye Lenses for Omnidirectional Coverage: The choice of ultra-wide FOV fisheye lenses, combined with a strategic camera arrangement, effectively achieves full spherical coverage with a minimal number of cameras.
- Neural Network for Robust Cost Computation: SweepNet's neural network architecture is designed to learn more discriminative features for matching, leading to cleaner cost volumes and more accurate depth predictions compared to conventional cost functions like ZNCC.
Key Results
- Outperforms State-of-the-Art: SweepNet consistently outperformed conventional methods like ZNCC and MC-CNN on synthetic datasets across various weather conditions, achieving lower Mean Absolute Error (MAE) and Root Mean Square (RMS) errors.
- Improved Accuracy with SGM: Combining SweepNet with SGM further enhanced performance, yielding the best and most robust results. For instance, SweepNet + SGM achieved an MAE of 4.8 and RMS of 1.1 on the Sunny dataset for errors greater than 5%.
- Accurate Reconstruction of Thin Structures: Qualitative results demonstrate successful reconstruction of challenging thin structures like traffic signs and poles, which are often problematic for stereo systems.
- Robustness to Varying Conditions: The system showed good performance on datasets with varying weather conditions, indicating its robustness.
Limitations and Future Work
- Limitations: The paper acknowledges that while their system is effective, the "disparity maps are in low resolution" and "high distortion by the fisheye lenses" can still pose challenges, particularly for vertical structures parallel to the baseline. The current system's depth estimation range is also tied to the baseline.
- Future Work: The authors suggest that their system is useful for drones that can move freely in all 6-DOF, implying potential for extending the system to full spherical depth estimation for aerial applications. The paper also mentions that the datasets and trained network will be made public, which will facilitate further research in this area.
Summary updated