Abstract
The integration of RGB and depth modalities significantly enhances the accuracy of segmenting complex indoor scenes, with depth data from RGB-D cameras playing a crucial role in this improvement. However, collecting an RGB-D dataset is more expensive than an RGB dataset due to the need for specialized depth sensors. Aligning depth and RGB images also poses challenges due to sensor positioning and issues like missing data and noise. In contrast, Pseudo Depth (PD) from high-precision depth estimation algorithms can eliminate the dependence on RGB-D sensors and alignment processes, as well as provide effective depth information and show significant potential in semantic segmentation. Therefore, to explore the practicality of utilizing pseudo depth instead of real depth for semantic segmentation, we design an RGB-PD segmentation pipeline to integrate RGB and pseudo depth and propose a Pseudo Depth Aggregation Module (PDAM) for fully exploiting the informative clues provided by the diverse pseudo depth maps. The PDAM aggregates multiple pseudo depth maps into a single modality, making it easily adaptable to other RGB-D segmentation methods. In addition, the pre-trained diffusion model serves as a strong feature extractor for RGB segmentation tasks, but multi-modal diffusion-based segmentation methods remain unexplored. Therefore, we present a Pseudo Depth Diffusion Model (PDDM) that adopts a large-scale text-image diffusion model as a feature extractor and a simple yet effective fusion strategy to integrate pseudo depth. To verify the applicability of pseudo depth and our PDDM, we perform extensive experiments on the NYUv2 and SUNRGB-D datasets. The experimental results demonstrate that pseudo depth can effectively enhance segmentation performance, and our PDDM achieves state-of-the-art performance, outperforming other methods by +6.98 mIoU on NYUv2 and +2.11 mIoU on SUNRGB-D.
1. Introduction
Semantic segmentation in complex indoor environments is a fundamental task for robotics and augmented reality. While RGB data provides rich texture information, depth (D) data offers critical geometric cues that help distinguish overlapping objects with similar appearances. Traditionally, this has relied on RGB-D cameras. However, physical depth sensors introduce significant overhead: they are expensive, require complex alignment with RGB sensors, and often produce noisy data with missing values in reflective or shadowed areas.
To overcome these limitations, we propose a shift from the traditional RGB-D pipeline to an RGB-PD (Pseudo Depth) pipeline. By leveraging high-precision monocular depth estimation algorithms, we can generate "pseudo depth" directly from RGB images, eliminating the need for specialized hardware and alignment processes.
2. Methodology
Our approach, the Pseudo Depth Diffusion Model (PDDM), introduces two primary innovations to make software-generated depth work effectively for segmentation.
2.1 Pseudo Depth Aggregation Module (PDAM)
Different depth estimation models (e.g., Depth Anything, MetaPrompt-SD, and Marigold) have varying strengths; some capture fine edges better, while others provide superior global consistency. The PDAM takes multiple pseudo-depth maps and fuses them into a single, high-quality modality. It employs spatial and channel attention mechanisms to dynamically weigh information from different sources, resulting in a fused depth map that is often cleaner and smoother than raw sensor data.
2.2 Pseudo Depth Diffusion Model (PDDM)
The core architecture utilizes a large-scale pre-trained Stable Diffusion UNet as a feature extractor. Typically, diffusion models work by removing Gaussian noise from an image. In PDDM, we treat the Pseudo Depth features as "structured noise." By injecting depth information into the diffusion process this way, the model naturally learns to "denoise" the scene using geometric clues. This strategy is highly efficient, adding only 0.37M extra parameters and negligible FLOPs (0.02G) compared to the baseline.
3. Experimental Results
We evaluated PDDM on two gold-standard datasets for indoor scene understanding: NYUv2 (1,449 images) and SUNRGB-D (over 10,000 images).
As shown in the results table, PDDM sets a new bar for indoor semantic segmentation. On NYUv2, it achieves 64.18 mIoU, outperforming the previous state-of-the-art (ODISE) by a significant margin. On SUNRGB-D, it reaches 55.11 mIoU, a +2.11 improvement over existing methods. Surprisingly, our RGB-PD approach often outperforms traditional RGB-D methods that use real sensor data, validating that high-quality pseudo depth can be more effective than noisy hardware scans.
4. Conclusion
PDDM demonstrates that pseudo depth is a viable and often superior alternative to physical depth sensors for semantic segmentation. By aggregating diverse depth estimations and integrating them into a diffusion-based backbone as structured noise, we achieve superior accuracy with minimal computational overhead. While the underlying diffusion backbone is computationally heavier than some real-time networks, the performance gains suggest significant potential for high-precision indoor scene understanding.
Summary updated