Unsupervised OmniMVS: Efficient Omnidirectional Depth Inference via Establishing Pseudo-Stereo Supervision

Zisong Chen, Chunyu Lin, Lang Nie, Kang Liao, Yao Zhao

Abstract

Omnidirectional Multi-View Stereo (MVS) is a critical component for 360° environmental perception in robotics and autonomous systems. However, existing deep-learning solutions rely heavily on supervised training with ground-truth depth data, which is difficult to acquire in diverse real-world settings. This paper presents Un-OmniMVS, the first unsupervised framework for 360° depth estimation using a multi-fisheye camera rig. By introducing a pseudo-stereo supervision mechanism and a frequency-domain attention module, the proposed method eliminates the need for labeled data while achieving superior generalization and a 2x increase in inference speed compared to supervised state-of-the-art models.

Introduction

Perceiving a full 360° field of view is essential for autonomous agents to navigate complex environments safely. While multi-fisheye camera systems provide the necessary hardware coverage, the software challenge remains significant. Current deep-learning models for omnidirectional MVS are predominantly supervised, requiring dense depth labels often generated via expensive LiDAR sensors or synthetic simulations. This reliance creates a "domain gap," where models trained on synthetic data fail to maintain accuracy when deployed in real-world scenarios with varying lighting and textures.

To address these challenges, we introduce Un-OmniMVS. Our approach shifts the paradigm from supervised learning to self-supervised geometry consistency. By leveraging the inherent redundancy in multi-view fisheye setups, the model learns to infer depth by ensuring photometric consistency across synthesized views, effectively bridging the gap between simulation and reality.

Methodology

The Un-OmniMVS architecture is built upon three primary technical innovations designed to optimize both accuracy and computational efficiency.

1. Pseudo-Stereo Supervision

The core of the unsupervised learning process is the "pseudo-stereo" pair generation. The system projects four raw fisheye images onto a virtual unit sphere located at the rig's center. From this spherical representation, the model generates two panoramic images in Equirectangular Projection (ERP) format. These panoramas represent the same scene but are rendered with a 90-degree relative rotation.

Because these two panoramas share a virtual center (zero-length baseline), the geometric relationship between them is defined purely by rotation and the underlying depth of the scene. The model is trained by enforcing photometric consistency: it predicts a depth map and then attempts to warp one panorama to match the other. The resulting reconstruction error serves as the supervisory signal, allowing the model to learn depth without any external labels.

2. Frequent-Spatial Feature Extractor

Traditional convolutional neural networks (CNNs) often struggle with global context, as standard pooling operations can discard fine spatial details. Un-OmniMVS incorporates a Frequency Attention Module (FAM) that utilizes Fast Fourier Transforms (FFT). This allows the network to capture non-local, global features in the frequency domain while preserving high-resolution local details in the spatial domain. This dual-domain approach is particularly effective for resolving depth in large, textureless regions like walls or ceilings.

3. Variance-Based Lightweight Cost Volume

Constructing a 3D cost volume for pixel matching is typically the most memory-intensive part of MVS. We implement a lightweight cost volume based on the variance between the synthetic pseudo-stereo panoramas. This design reduces the memory footprint to approximately 25% of previous supervised methods, enabling faster processing and deployment on resource-constrained hardware.

Experimental Evaluation

Datasets

The model was rigorously tested on three benchmark synthetic datasets:

  • OmniThings: A large-scale dataset featuring over 10,000 diverse synthetic scenes.
  • OmniHouse: Realistic indoor environments containing over 2,500 distinct house models.
  • Sunny: An outdoor street scene dataset used to test performance in natural lighting.

To validate real-world utility, the authors also deployed the model on a physical hardware rig equipped with four 220° FoV fisheye lenses.

Results and Performance

Un-OmniMVS demonstrates significant improvements in both efficiency and robustness:

  • Inference Speed: The model achieves an inference time of 0.33s per frame, which is twice as fast as the supervised OmniMVS (0.67s).
  • Accuracy: On the "Sunny" outdoor dataset, the unsupervised model outperformed the supervised state-of-the-art, achieving a Mean Absolute Error (MAE) of 1.6 compared to the supervised model's 1.9.
  • Generalization: In cross-dataset evaluations, Un-OmniMVS showed significantly lower error rates than supervised models. This suggests that by learning the fundamental geometry of light and rotation rather than memorizing synthetic patterns, the model is far more robust when encountering unfamiliar environments.

Discussion and Future Work

While Un-OmniMVS represents a breakthrough in unsupervised omnidirectional sensing, certain limitations remain. Hardware memory constraints currently limit the number of depth samples to 32 during real-world testing; increasing this resolution would likely yield finer depth precision. Additionally, the model's reliance on photometric consistency can be challenged by chromatic aberration—color shifts between different camera lenses. While a "Gradient Loss" function was implemented to mitigate this, developing more robust color-invariant matching remains a priority.

Finally, the current framework assumes a static environment. Future iterations will aim to incorporate temporal consistency to better handle dynamic objects in the unsupervised training loop, further enhancing the system's applicability to high-speed autonomous navigation.

Conclusion

Un-OmniMVS proves that supervised labels are not a prerequisite for high-quality omnidirectional depth estimation. By establishing pseudo-stereo supervision and utilizing frequency-domain features, we provide a path toward more efficient, generalizable, and easily deployable 3D perception systems for the next generation of autonomous robots.

Summary updated

BibTeX
@inproceedings{chen2023unsupervised,
  author = {Zisong Chen and Chunyu Lin and Lang Nie and Kang Liao and Yao Zhao},
  title = {{Unsupervised OmniMVS: Efficient Omnidirectional Depth Inference via Establishing Pseudo-Stereo Supervision}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342332},
}
Figure