OmniDiffusion: Reformulating 360 Monocular Depth Estimation using Semantic and Surface Normal Conditioned Diffusion

Payal Mohadikar, Ye DuanWACV 2025

TL;DR

This paper introduces OmniDiffusion, a novel framework that reformulates 360 monocular depth estimation as a diffusion denoising process. By conditioning the diffusion model on RGB images, semantic segmentation, and surface normals, OmniDiffusion achieves state-of-the-art results in generating accurate, detailed, and globally consistent depth maps from omnidirectional images.

What Problem Does This Paper Address?

Estimating depth from a single image (monocular depth estimation or MDE) is a fundamental but challenging computer vision task. While traditional MDE methods often struggle with limited field-of-view, 360-degree cameras offer a comprehensive scene perspective. However, the common Equirectangular Projection (ERP) format for 360 images introduces distortions, particularly in polar regions, making accurate depth estimation difficult. Existing methods, including those using CNNs, transformers, and even recent diffusion models, often suffer from global inconsistencies, poor detail, and limited generalization when applied to 360-degree imagery. This paper aims to overcome these limitations by leveraging the generative power of diffusion models in a novel way for 360 MDE.

Methods

The core of OmniDiffusion is a latent diffusion model that treats depth estimation as an iterative denoising process. Instead of directly predicting depth, the model learns to denoise random depth distributions into accurate depth maps.

Key methodological contributions include:

  • Diffusion Denoising Reformulation: The 360 MDE task is framed as reversing a diffusion process, where noisy depth distributions are progressively refined into coherent depth maps.
  • Latent Space Diffusion: To improve efficiency, the diffusion and denoising processes are performed in a learned latent space, reducing computational costs.
  • Multi-Modal Conditioning: The denoising process is guided by multiple visual cues:
  • RGB Image: The primary visual input.
  • Semantic Segmentation: Provides object-level contextual information.
  • Surface Normals: Offers geometric constraints, crucial for sharp details and discontinuities. These conditions are encoded into a unified latent representation.

  • Transformer-based Feature Extractor: A Swin Transformer is employed to extract rich features from the visual conditions, demonstrating superior performance compared to standard CNNs like ResNet.

  • Depth Encoder-Decoder: A learned depth encoder maps ground truth depth to the latent space, and a depth decoder reconstructs the final 360-degree depth map from the denoised latent representation.

Datasets

The paper uses the following datasets for training and evaluation:

  • Stanford2D3D: Used for training and evaluation, particularly for quantitative comparisons.
  • Matterport3D: Used for training and evaluation, including cross-dataset generalization tests.

Key Insights and Ideas

  • Diffusion Models for 360 MDE: This paper is the first to explore diffusion models for 360 monocular depth estimation, demonstrating their potential beyond image synthesis.
  • Synergistic Conditioning: Combining RGB, semantic segmentation, and surface normal information as conditions significantly enhances depth estimation accuracy and detail. Surface normals, in particular, are shown to be crucial for preserving sharp geometric discontinuities.
  • Latent Space Efficiency: Performing diffusion in a latent space is key to making the computationally intensive diffusion process practical for high-resolution depth estimation.
  • Transformer's Advantage: The choice of a transformer-based feature extractor for visual conditions proves more effective than traditional CNNs for capturing complex relationships in the input data.

Key Results

  • State-of-the-Art Performance: OmniDiffusion achieves state-of-the-art (SOTA) results on the Stanford2D3D and Matterport3D datasets across various metrics, including Abs Rel and RMSE.
  • Quantitative Improvements:
    • On Stanford2D3D, OmniDiffusion achieves an Abs Rel of 0.0889 and RMSE of 0.3285, outperforming previous SOTA methods like OmniFusion (0.0943 Abs Rel, 0.3582 RMSE).
  • On Matterport3D, it achieves an Abs Rel of 0.1060 and RMSE of 0.4783.
  • Improved Detail and Consistency: Qualitative results show that OmniDiffusion generates sharper object boundaries, more detailed structures, and globally consistent depth maps compared to existing methods, which often exhibit blocky artifacts or local inconsistencies.
  • Strong Generalization: The model demonstrates robust generalization capabilities, outperforming existing omnidirectional MDE methods in cross-dataset evaluations.

Limitations and Future Work

  • Computational Cost: While latent diffusion improves efficiency, diffusion models are inherently computationally intensive. The authors mention that further research is needed to improve their efficiency, especially for real-time applications.
  • Inference Steps: The performance is sensitive to the number of inference steps, with 20 steps showing optimal results. Increasing steps improves performance but at a higher computational cost.
  • Future Directions: The paper suggests exploring further improvements in efficiency and potentially extending the framework to video depth estimation. The agnostic nature of the framework to image encoders also opens avenues for experimenting with newer architectures.

Summary updated

BibTeX
@inproceedings{mohadikar2025omnidiffusion,
  author = {Payal Mohadikar and Ye Duan},
  title = {{OmniDiffusion: Reformulating 360 Monocular Depth Estimation using Semantic and Surface Normal Conditioned Diffusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure