Introduction
Estimating depth from a single 360° image—monocular depth estimation—is a critical capability for applications in virtual reality (VR), augmented reality (AR), and robotics. However, this task faces two significant hurdles: chronic data scarcity and inherent geometric distortion. Capturing ground truth depth for 360° scenes is difficult because active sensors like LiDAR often appear within the camera's own field of view. Furthermore, 360° images are typically stored in an equirectangular projection (ERP), which introduces heavy distortion at the poles and challenges standard Convolutional Neural Networks (CNNs) that possess limited receptive fields.
Proposed Methodology
To address these challenges, this research introduces a novel framework combining a transformer-based architecture with a joint training scheme.
1. Joint Supervised and Self-Supervised Learning
The authors propose a hybrid objective function to leverage the strengths of different learning paradigms:
- Supervised Flow: Utilizes available ground truth data to provide a "grounding" for the model. This helps the network avoid common self-supervised pitfalls, such as misinterpreting light reflections or transparent surfaces as complex geometry.
- Self-Supervised Flow: Employs gravity-aligned video sequences to enforce image, depth, and pose consistency. By assuming gravity alignment, the authors simplify camera motion into four variables, stabilizing the learning process.
2. Non-local Dense Prediction Transformer (DPT)
Moving beyond local kernels, the model utilizes a Vision Transformer (ViT) backbone. Transformers are inherently better at capturing global context through attention mechanisms. The architecture is further enhanced by a Non-local Fusion Block (NLFB) during the upsampling process. This allows the network to maintain a global outlook, ensuring that large-scale structures—like a wall spanning the entire horizontal axis of a 360° view—are recognized as continuous objects.
3. Cross-Domain Transfer Learning
To mitigate the lack of 360° data, the model is pre-trained on large-scale rectilinear (standard) image datasets. While the projections differ, the relative depth relationships remain consistent across domains, providing a robust starting point for the transformer.
Experimental Results
The proposed method was evaluated on key benchmarks including 3D60 and Stanford (2D-3D-S), establishing a new state-of-the-art (SOTA). On the Stanford3D dataset, the model achieved an Absolute Relative Error (AbsRel) of 0.0649, significantly outperforming previous leaders like HoHoNet (0.0901).
Qualitative results demonstrate that the model is superior at distinguishing complex indoor features. For instance, it correctly identifies windows as part of a flat wall rather than "holes" in the geometry—a common error in models trained solely with supervised data.
Conclusion and Future Work
By combining the global receptive field of Transformers with a joint learning objective, this approach effectively overcomes the geometric and data-related limitations of 360° monocular depth estimation. While the current system relies on gravity-aligned video for its self-supervised component, future work may explore more flexible pose estimation to handle uncalibrated handheld captures. The success of the non-local fusion strategy also suggests potential applications in other panoramic tasks, such as semantic segmentation and scene layout estimation.
Summary updated