PDDM: Pseudo Depth Diffusion Model for RGB-PD Semantic Segmentation Based in Complex Indoor Scenes

Xinhua Xu, Hong Liu, Jianbing Wu, Jinfu Liu

TL;DR

PDDM (Pseudo Depth Diffusion Model) eliminates the need for expensive RGB-D sensors by using "pseudo depth" estimated directly from RGB images for semantic segmentation. By aggregating multiple depth estimations and injecting them into a pre-trained diffusion model as "structured noise," the authors achieve new state-of-the-art results on major indoor scene benchmarks.

What Problem Does This Paper Address?

Semantic segmentation in complex indoor scenes is challenging because objects often overlap or share similar colors. While adding a depth (D) channel to standard RGB data helps significantly, it comes with three major headaches:

  1. Cost: RGB-D cameras are more expensive and less common than standard cameras.
  2. Alignment: Aligning depth maps with RGB images is difficult due to different sensor positions.
  3. Data Quality: Raw depth data is often "noisy," containing holes (missing data) or rough edges caused by reflections and shadows.

The authors ask: Can we get the benefits of depth without the hardware? They propose using Pseudo Depth (PD)—depth maps generated by software algorithms from a single RGB image—to replace physical sensors.

Methods

The researchers introduce two primary innovations to make "Pseudo Depth" work effectively:

1. Pseudo Depth Aggregation Module (PDAM)

Not all depth estimation models are created equal. Some excel at edges, while others are better at global geometry. PDAM takes multiple pseudo-depth maps (generated by different "meta" models) and fuses them. It uses spatial and channel attention to weigh the most reliable information from each source, creating a single, high-quality aggregated depth map that is often cleaner and smoother than real sensor data.

2. Pseudo Depth Diffusion Model (PDDM)

The core architecture leverages a large-scale pre-trained Stable Diffusion UNet as a feature extractor. Typically, diffusion models work by removing Gaussian noise from an image.

  • The "Aha!" Moment: Instead of adding random Gaussian noise to the RGB features, the authors treat the Pseudo Depth features as "structured noise."
  • By injecting depth information into the diffusion process this way, the model naturally learns to "denoise" the scene by using geometric clues, effectively fusing the two modalities without the massive computational overhead of a traditional dual-branch network.

Datasets

The model was evaluated on two gold-standard datasets for indoor scene understanding:

  • NYUv2: 1,449 indoor images across 40 semantic classes.
  • SUNRGB-D: A larger-scale dataset with over 10,000 images and 37 classes.
  • Meta-Depth Sources: To generate the pseudo depth, they used three leading models: Depth Anything, MetaPrompt-SD, and Marigold.

Key Insights and Ideas

  • Pseudo Depth > Real Depth: Surprisingly, the authors found that pseudo depth can sometimes outperform real depth. Because pseudo depth is generated by models trained on massive datasets, it is often smoother and lacks the "holes" and sensor noise found in raw hardware scans.
  • Diversity is Strength: Aggregating three different depth models yielded better results than using any single one. This suggests that different depth estimation architectures provide complementary geometric "views" of a scene.
  • Efficiency through Diffusion: By treating depth as noise in a pre-trained diffusion backbone, they added only 0.37M extra parameters and negligible FLOPs (0.02G) compared to the baseline, making it much more efficient than previous dual-branch fusion methods.

Key Results

PDDM sets a new bar for indoor semantic segmentation:

  • NYUv2: Achieved 64.18 mIoU, outperforming the previous state-of-the-art (ODISE) by a significant margin.
  • SUNRGB-D: Achieved 55.11 mIoU, a +2.11 improvement over existing methods.
  • Ablation Success: The tests showed that even a simple version of their model (using only one pseudo-depth source) still outperformed most traditional RGB-D models that use real sensor data.

Limitations and Future Work

  • Inference Speed: While the authors optimized the fusion strategy, the underlying diffusion backbone is still computationally heavier than lightweight real-time networks (like SegFormer).
  • Dependency on Meta-Models: The performance of PDDM is inherently tied to the quality of the monocular depth estimation models used. As those models improve, PDDM will likely improve, but it remains a "downstream" dependency.
  • Future Directions: The authors suggest exploring how this "depth-as-noise" strategy might work in outdoor autonomous driving scenarios or for other dense prediction tasks like instance segmentation.

Summary updated

Abstract

The integration of RGB and depth modalities significantly enhances the accuracy of segmenting complex indoor scenes, with depth data from RGB-D cameras playing a crucial role in this improvement. However, collecting an RGB-D dataset is more expensive than an RGB dataset due to the need for specialized depth sensors. Aligning depth and RGB images also poses challenges due to sensor positioning and issues like missing data and noise. In contrast, Pseudo Depth (PD) from high-precision depth estimation algorithms can eliminate the dependence on RGB-D sensors and alignment processes, as well as provide effective depth information and show significant potential in semantic segmentation. Therefore, to explore the practicality of utilizing pseudo depth instead of real depth for semantic segmentation, we design an RGB-PD segmentation pipeline to integrate RGB and pseudo depth and propose a Pseudo Depth Aggregation Module (PDAM) for fully exploiting the informative clues provided by the diverse pseudo depth maps. The PDAM aggregates multiple pseudo depth maps into a single modality, making it easily adaptable to other RGB-D segmentation methods. In addition, the pre-trained diffusion model serves as a strong feature extractor for RGB segmentation tasks, but multi-modal diffusion-based segmentation methods remain unexplored. Therefore, we present a Pseudo Depth Diffusion Model (PDDM) that adopts a large-scale text-image diffusion model as a feature extractor and a simple yet effective fusion strategy to integrate pseudo depth. To verify the applicability of pseudo depth and our PDDM, we perform extensive experiments on the NYUv2 and SUNRGB-D datasets. The experimental results demonstrate that pseudo depth can effectively enhance segmentation performance, and our PDDM achieves state-of-the-art performance, outperforming other methods by +6.98 mIoU on NYUv2 and +2.11 mIoU on SUNRGB-D.
BibTeX
@inproceedings{xu2025pddm,
  author = {Xinhua Xu and Hong Liu and Jianbing Wu and Jinfu Liu},
  title = {{PDDM: Pseudo Depth Diffusion Model for RGB-PD Semantic Segmentation Based in Complex Indoor Scenes}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.32970},
}

Loading related content...

Figure