Abstract
Monocular depth estimation is a challenging task that predicts the pixel-wise depth from a single 2D image. Current methods typically model this problem as a regression or classification task. We propose DiffusionDepth, a new approach that reformulates monocular depth estimation as a denoising diffusion process. It learns an iterative denoising process to 'denoise' random depth distribution into a depth map with the guidance of monocular visual conditions. The process is performed in the latent space encoded by a dedicated depth encoder and decoder. Instead of diffusing ground truth (GT) depth, the model learns to reverse the process of diffusing the refined depth of itself into random depth distribution. This self-diffusion formulation overcomes the difficulty of applying generative models to sparse GT depth scenarios. The proposed approach benefits this task by refining depth estimation step by step, which is superior for generating accurate and highly detailed depth maps. Experimental results on KITTI and NYU-Depth-V2 datasets suggest that a simple yet efficient diffusion approach could reach state-of-the-art performance in both indoor and outdoor scenarios with acceptable inference time.
Introduction
Monocular Depth Estimation (MDE)—predicting 3D distance from a single 2D image—is an inherently "ill-posed" problem because a single 2D point could represent many points in 3D space. Current mainstream approaches treat this as a regression problem (predicting a continuous value per pixel) or a classification problem (sorting pixels into distance "bins"). However, regression often leads to over-smoothed results and lost details, while bin-based classification can create "staircase" artifacts and blurriness at object boundaries.
Furthermore, while generative models like Denoising Diffusion Probabilistic Models (DDPMs) are excellent at creating high-quality images, they struggle with depth datasets like KITTI, where only about 5% of the pixels have ground-truth labels. This sparsity usually causes generative models to "collapse" or fail to learn the global structure of a scene.
Methods
DiffusionDepth applies the logic of diffusion models to depth estimation through several key innovations:
- Latent Space Diffusion: Instead of diffusing raw, high-resolution depth maps, the model uses a dedicated encoder to compress depth into a "latent space." The denoising happens here, and a decoder brings it back to full resolution, significantly reducing computational costs.
- Self-Diffusion Formulation: To handle sparse ground truth (GT), the model doesn't just diffuse the GT. Instead, it adds noise to its own refined depth predictions during training. It then learns to reverse that noise while being supervised by the sparse GT. This forces the model to maintain structural consistency across the entire image, even where data is missing.
- Monocular Guidance: The denoising process is conditioned on the input RGB image. The model uses a Swin Transformer backbone and a Feature Pyramid Network (FPN) to extract multi-scale visual features, which guide the denoising blocks (MCDB) to place objects at the correct depths.
Key Results and Visualization
The model was evaluated on two industry-standard benchmarks: KITTI (outdoor driving) and NYU-Depth-V2 (indoor). DiffusionDepth achieved state-of-the-art performance, reaching an RMSE of 2.016 on the KITTI Eigen split and 0.295 on NYU-Depth-V2.
Beyond numerical metrics, the iterative refinement process allows the model to "sculpt" depth maps that are much cleaner and more realistic than traditional regression models. Early steps establish the coarse layout of the scene, while later steps sharpen the edges of furniture or thin structures like signposts.
To make the model practical for real-world use, the authors employ DDIM sampling, allowing for high-quality generation in just 20 steps. Using a ResNet backbone, the model achieves approximately 14 FPS on an RTX 3090, making it viable for many near-real-time applications.
Conclusion
DiffusionDepth demonstrates that reformulating depth estimation as a denoising process offers significant advantages in detail preservation and accuracy. The self-diffusion strategy effectively bridges the gap between generative modeling and sparse data scenarios. Future work may explore the probabilistic nature of these models to provide uncertainty estimation, which is critical for safety-sensitive tasks like autonomous robotics.
Summary updated