Abstract
This paper presents an upright and stabilized omnidirectional depth estimation for an arbitrarily rotated wide-baseline multi-camera inertial system. By aligning the reference rig coordinate system with the gravity direction acquired from an inertial measurement unit, we sample depth hypotheses for omnidirectional stereo matching by sweeping global spheres whose equators are parallel to the ground plane. Then, unary features extracted from each input image by 2D convolutional neural networks (CNN) are warped onto the swept spheres, and the final omnidirectional depth map is output through cost computation by a 3D CNN-based hourglass module and a softargmax operation. This can eliminate wavy or unrecognizable visual artifacts in equirectangular depth maps which can cause failures in scene understanding. We show the capability of our upright and stabilized omnidirectional depth estimation through experiments on real data.
Introduction
Omnidirectional (360-degree) vision is a cornerstone for autonomous navigation, providing a comprehensive view of the environment. However, standard wide-baseline multi-camera systems often suffer from significant distortions when the camera rig is not perfectly level. In real-world applications—such as drones or handheld devices—tilting and shaking cause "wavy" artifacts in equirectangular depth maps. These distortions are not merely visual; they degrade the performance of downstream tasks like object detection and semantic segmentation by warping the geometric features of the scene beyond what neural networks are typically trained to recognize.
Methodology: Upright Spherical Sweeping
The core innovation of this work is the Upright Spherical Sweeping algorithm, which integrates Inertial Measurement Unit (IMU) data directly into the depth estimation pipeline. Instead of post-processing a distorted image, the system corrects the distortion during the sampling process.
- Gravity Alignment: The system utilizes an IMU to determine the gravity vector. By aligning the reference rig coordinate system with gravity, the system ensures that depth hypotheses are sampled along global spheres whose equators remain parallel to the ground plane.
- 2-DOF Correction: The algorithm specifically corrects for Roll and Pitch rotations, which are the primary causes of horizon distortion, while maintaining the original Yaw to preserve the camera's heading.
- CNN-based Pipeline: Visual features are extracted from four 220° FOV fisheye cameras using a 2D CNN. These features are then warped onto the "upright" spheres. A 3D CNN-based hourglass module processes the resulting cost volume to find the best depth match for every pixel, followed by a
softargmaxoperation to produce the final stabilized depth map.
Experimental Results
The system was evaluated using a custom square rig (0.3m x 0.3m) equipped with four fisheye cameras and an Xsens MTi-10 IMU.
Qualitative Improvements
As shown in the visual comparisons, the proposed method effectively eliminates the "funhouse mirror" effect seen in traditional unstabilized depth maps. In corridor and office sequences, walls and floors appear straight and correctly oriented despite the rig's physical tilt. This stabilization makes the output significantly more usable for both human operators and automated recognition systems.
Trajectory and Stability
The robustness of the gravity-aligned approach is further evidenced by the system's ability to maintain a stable orientation relative to the gravity vector throughout complex movements. Trajectory analysis confirms that the estimated rig poses remained well-aligned with the gravity direction throughout the test sequences.
Conclusion
By shifting stabilization from post-processing to the sampling stage of the depth estimation pipeline, this research provides a robust solution for omnidirectional vision on dynamic platforms. The resulting "upright" depth maps are significantly more reliable for autonomous scene understanding. Future work may involve integrating this stabilized depth into full SLAM systems and testing under high-acceleration scenarios where gravity estimation becomes more challenging.
Summary updated