Upright and Stabilized Omnidirectional Depth Estimation for Wide-baseline Multi-camera Inertial Systems

Changhee Won, Hochang Seok, Jongwoo LimWorkshops @ CVPR 2020 · OpenReview

Abstract

This paper presents an upright and stabilized omnidirectional depth estimation for an arbitrarily rotated wide-baseline multi-camera inertial system. By aligning the reference rig coordinate system with the gravity direction acquired from an inertial measurement unit, we sample depth hypotheses for omnidirectional stereo matching by sweeping global spheres whose equators are parallel to the ground plane. Then, unary features extracted from each input image by 2D convolutional neural networks (CNN) are warped onto the swept spheres, and the final omnidirectional depth map is output through cost computation by a 3D CNN-based hourglass module and a softargmax operation. This can eliminate wavy or unrecognizable visual artifacts in equirectangular depth maps which can cause failures in scene understanding. We show the capability of our upright and stabilized omnidirectional depth estimation through experiments on real data.

Introduction

Omnidirectional (360-degree) vision is a cornerstone for autonomous navigation, providing a comprehensive view of the environment. However, standard wide-baseline multi-camera systems often suffer from significant distortions when the camera rig is not perfectly level. In real-world applications—such as drones or handheld devices—tilting and shaking cause "wavy" artifacts in equirectangular depth maps. These distortions are not merely visual; they degrade the performance of downstream tasks like object detection and semantic segmentation by warping the geometric features of the scene beyond what neural networks are typically trained to recognize.

Methodology: Upright Spherical Sweeping

The core innovation of this work is the Upright Spherical Sweeping algorithm, which integrates Inertial Measurement Unit (IMU) data directly into the depth estimation pipeline. Instead of post-processing a distorted image, the system corrects the distortion during the sampling process.

  1. Gravity Alignment: The system utilizes an IMU to determine the gravity vector. By aligning the reference rig coordinate system with gravity, the system ensures that depth hypotheses are sampled along global spheres whose equators remain parallel to the ground plane.
  2. 2-DOF Correction: The algorithm specifically corrects for Roll and Pitch rotations, which are the primary causes of horizon distortion, while maintaining the original Yaw to preserve the camera's heading.
  3. CNN-based Pipeline: Visual features are extracted from four 220° FOV fisheye cameras using a 2D CNN. These features are then warped onto the "upright" spheres. A 3D CNN-based hourglass module processes the resulting cost volume to find the best depth match for every pixel, followed by a softargmax operation to produce the final stabilized depth map.
Comparison of unstabilized (middle) vs. stabilized (right) equirectangular RGB and depth maps across different environments.

Experimental Results

The system was evaluated using a custom square rig (0.3m x 0.3m) equipped with four fisheye cameras and an Xsens MTi-10 IMU.

Qualitative Improvements

As shown in the visual comparisons, the proposed method effectively eliminates the "funhouse mirror" effect seen in traditional unstabilized depth maps. In corridor and office sequences, walls and floors appear straight and correctly oriented despite the rig's physical tilt. This stabilization makes the output significantly more usable for both human operators and automated recognition systems.

Trajectory and Stability

The robustness of the gravity-aligned approach is further evidenced by the system's ability to maintain a stable orientation relative to the gravity vector throughout complex movements. Trajectory analysis confirms that the estimated rig poses remained well-aligned with the gravity direction throughout the test sequences.

Estimated rig trajectory showing alignment between the camera orientation (orange) and the gravity vector (blue).

Conclusion

By shifting stabilization from post-processing to the sampling stage of the depth estimation pipeline, this research provides a robust solution for omnidirectional vision on dynamic platforms. The resulting "upright" depth maps are significantly more reliable for autonomous scene understanding. Future work may involve integrating this stabilized depth into full SLAM systems and testing under high-acceleration scenarios where gravity estimation becomes more challenging.

Summary updated

Abstract

This paper presents an upright and stabilized omnidirectional depth estimation for an arbitrarily rotated wide-baseline multi-camera inertial system. By aligning the reference rig coordinate system with the gravity direction acquired from an inertial measurement unit, we sample depth hypotheses for omnidirectional stereo matching by sweeping global spheres whose equators are parallel to the ground plane. Then, unary features extracted from each input image by 2D convolutional neural networks (CNN) are warped onto the swept spheres, and the final omnidirectional depth map is output through cost computation by a 3D CNN-based hourglass module and a softargmax operation. This can eliminate wavy or unrecognizable visual artifacts in equirectangular depth maps which can cause failures in scene understanding. We show the capability of our upright and stabilized omnidirectional depth estimation through experiments on real data.
BibTeX
@inproceedings{won2020upright,
  author = {Changhee Won and Hochang Seok and Jongwoo Lim},
  title = {{Upright and Stabilized Omnidirectional Depth Estimation for Wide-baseline Multi-camera Inertial Systems}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=bmPLNiM-5P},
}

Loading related content...

Figure