Depth-guided Texture Diffusion for Image Semantic Segmentation

Wei Sun, Yuan Li, Qixiang Ye, Jianbin Jiao, Yanzhao Zhou

Abstract

Depth information provides valuable insights into the 3D structure especially the outline of objects, which can be utilized to improve the semantic segmentation tasks. However, a naive fusion of depth information can disrupt feature and compromise accuracy due to the modality gap between the depth and the vision. In this work, we introduce a Depth-guided Texture Diffusion approach that effectively tackles the outlined challenge. Our method extracts low-level features from edges and textures to create a texture image. This image is then selectively diffused across the depth map, enhancing structural information vital for precisely extracting object outlines. By integrating this enriched depth map with the original RGB image into a joint feature embedding, our method effectively bridges the disparity between the depth map and the image, enabling more accurate semantic segmentation. We conduct comprehensive experiments across diverse, commonly-used datasets spanning a wide range of semantic segmentation tasks, including Camouflaged Object Detection (COD), Salient Object Detection (SOD), and indoor semantic segmentation. With source-free estimated depth or depth captured by depth cameras, our method consistently outperforms existing baselines and achieves new state-of-the-art results.

Introduction

In computer vision, combining RGB images with depth data (RGB-D) is a powerful way to understand 3D structures and object boundaries. However, a fundamental modality gap exists: RGB images are rich in color and fine texture, while depth maps (whether from sensors or estimation models) are often "texture-less," representing only broad spatial relationships and distances.

When depth data is fused "naively" (e.g., channel stacking), the lack of detail in the depth map can disrupt high-quality RGB features, leading to decreased accuracy. This is particularly problematic in Camouflaged Object Detection (COD), where objects match background textures, and Salient Object Detection (SOD), where precise boundaries are critical. The core insight of this work is that texture can serve as a bridge: by making the depth map "look" more like the RGB image in terms of local detail, the fusion process becomes more stable and effective.

Methodology

The proposed framework harmonizes these modalities through a three-stage process:

  1. Texture Extraction (TE): The model uses a Fast Fourier Transform (FFT) to move the RGB image into the frequency domain. A high-pass filter isolates fine edges and textures, which are then converted back into a "texture image" ($X_h$).
  2. Texture Diffusion (TXD): This core innovation treats depth map pixels as nodes in a graph. Extracted RGB textures predict "diffusion weights" that guide an iterative message-passing process, effectively "painting" fine RGB textures onto the depth map skeleton.
  3. Structural Consistency Optimization (SCO): To ensure geometry is preserved, a structural loss function based on the Structural Similarity Index (SSIM) keeps the enriched depth map aligned with the original RGB image's structure.
  4. Joint Embedding & Adaptor: The enriched depth and RGB images are combined via element-wise addition and fed into a "Joint Embedding" module. This is integrated into backbones (like HitNet or DFormer) using "Adaptors" that inject depth information into every layer without altering the fundamental architecture.
Overview of the Depth-guided Texture Diffusion framework.

Experimental Results

The method was evaluated across three tasks: Salient Object Detection (NLPR, NJUK, STERE, SIP), Camouflaged Object Detection (CAMO, CHAMELEON, COD10K, NC4K), and Indoor Semantic Segmentation (NYUDepthv2, SUN-RGBD). Notably, for COD, "source-free" estimated depth was used, proving the method's utility even without high-quality sensors.

Quantitative Performance

The proposed method achieved new state-of-the-art (SOTA) results:

  • Camouflaged Object Detection: On the CAMO dataset, the model improved the $F_\beta$ (F-measure) by 9.1% and the $E_\xi$ (Enhanced-alignment measure) by 9.5% over previous methods.
  • Salient Object Detection: Consistently outperformed existing baselines like PopNet across all datasets.
  • Indoor Segmentation: On NYUDepthv2, the model reached a 58.0% mIoU, a significant jump over the 56.1% achieved by the baseline DFormer-L.

Qualitative Analysis

Visual results demonstrate superior performance in capturing thin structures, such as the antennae and legs of camouflaged shrimp, which are typically lost in standard depth-fusion models.

Qualitative comparison of the proposed method against various baselines on challenging segmentation tasks.

Conclusion

This work demonstrates that "texture-diffusion" effectively bridges the modality gap between RGB and depth data. While the iterative diffusion process adds some computational complexity, the gains in segmentation accuracy—especially for fine-grained structures—are significant. Future work may extend this philosophy to other multi-modal tasks, such as medical imaging or infrared-visible light fusion.

Summary updated

Abstract

Depth information provides valuable insights into the 3D structure especially the outline of objects, which can be utilized to improve the semantic segmentation tasks. However, a naive fusion of depth information can disrupt feature and compromise accuracy due to the modality gap between the depth and the vision. In this work, we introduce a Depth-guided Texture Diffusion approach that effectively tackles the outlined challenge. Our method extracts low-level features from edges and textures to create a texture image. This image is then selectively diffused across the depth map, enhancing structural information vital for precisely extracting object outlines. By integrating this enriched depth map with the original RGB image into a joint feature embedding, our method effectively bridges the disparity between the depth map and the image, enabling more accurate semantic segmentation. We conduct comprehensive experiments across diverse, commonly-used datasets spanning a wide range of semantic segmentation tasks, including Camouflaged Object Detection (COD), Salient Object Detection (SOD), and indoor semantic segmentation. With source-free estimated depth or depth captured by depth cameras, our method consistently outperforms existing baselines and achieves new state-of-theart results, demonstrating the effectiveness of our Depth-guided Texture Diffusion for image semantic segmentation.
BibTeX
@article{sun2025depth,
  author = {Wei Sun and Yuan Li and Qixiang Ye and Jianbin Jiao and Yanzhao Zhou},
  title = {{Depth-guided Texture Diffusion for Image Semantic Segmentation}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
  url = {https://openreview.net/forum?id=9nykXYt01s},
}
Figure