TL;DR
This paper introduces a Depth-guided Texture Diffusion approach to bridge the "modality gap" between RGB images and depth maps. By extracting high-frequency textures from RGB data and iteratively diffusing them into depth maps, the authors create a "texture-enriched" depth representation that significantly improves performance in salient, camouflaged, and indoor object segmentation.
What Problem Does This Paper Address?
In computer vision, combining RGB images with depth data (RGB-D) is a powerful way to understand 3D structures and object boundaries. However, there is a fundamental modality gap: RGB images are rich in color and fine texture, while depth maps (whether from sensors or estimation models) are often "texture-less," representing only broad spatial relationships and distances.
When researchers try to fuse these two types of data "naively" (e.g., just stacking them as channels), the lack of detail in the depth map can actually disrupt the high-quality features in the RGB image, leading to less accurate segmentation. This is particularly problematic in Camouflaged Object Detection (COD), where an object perfectly matches its background's texture, and Salient Object Detection (SOD), where precise boundaries are critical.
Methods
The authors propose a three-stage framework to harmonize these modalities:
- Texture Extraction (TE): Instead of working in the standard pixel space, the model uses a Fast Fourier Transform (FFT) to move the RGB image into the frequency domain. It applies a high-pass filter to isolate fine edges and textures, then converts this back into a "texture image" ($X_h$).
- Texture Diffusion (TXD): This is the core innovation. The model treats pixels in the depth map as nodes in a graph. It uses the extracted RGB textures to predict "diffusion weights," which guide an iterative message-passing process. This effectively "paints" the fine RGB textures onto the depth map skeleton, creating a texture-enriched depth map.
- Structural Consistency Optimization (SCO): To ensure the diffusion process doesn't warp the actual geometry of the scene, the authors use a structural loss function based on the Structural Similarity Index (SSIM). This keeps the enriched depth map aligned with the original RGB image's structure.
- Joint Embedding & Adaptor: Finally, the enriched depth and RGB images are combined via element-wise addition and fed into a "Joint Embedding" module. This is then integrated into standard backbones (like HitNet or DFormer) using "Adaptors" that inject depth information into every layer of the network without changing its fundamental architecture.
Datasets
The researchers evaluated their method across three distinct tasks using several standard benchmarks:
- Salient Object Detection (SOD): NLPR, NJUK, STERE, and SIP.
- Camouflaged Object Detection (COD): CAMO, CHAMELEON, COD10K, and NC4K.
- Indoor Semantic Segmentation: NYUDepthv2 and SUN-RGBD.
Notably, for the COD task, they used "source-free" estimated depth (generated by a pre-trained model) rather than ground-truth sensor data, proving the method works even when high-quality depth sensors aren't available.
Key Insights and Ideas
- Texture as the Bridge: The "aha moment" here is realizing that texture is the missing link. By making the depth map "look" more like the RGB image in terms of local detail, the fusion process becomes much more stable and effective.
- Frequency Domain Advantage: Extracting textures via FFT is more robust than simple edge detection because it allows for precise control over which frequencies (details) are passed to the depth map.
- Iterative Refinement: The diffusion process isn't a one-shot deal. The authors found that performance improves as the texture "spreads" through the depth map, typically saturating around four iterations.
- Architecture Agnostic: Because the depth information is injected via "Adaptors," this method can be bolted onto various state-of-the-art backbones (like Transformers or CNNs) with minimal modification.
Key Results
The proposed method achieved new state-of-the-art (SOTA) results across the board:
- Camouflaged Object Detection: On the CAMO dataset, the model improved the $F_\beta$ (F-measure) by 9.1% and the $E_\xi$ (Enhanced-alignment measure) by 9.5% over previous methods.
- Salient Object Detection: Consistently outperformed existing baselines like PopNet across all four datasets.
- Indoor Segmentation: On NYUDepthv2, the model reached a 58.0% mIoU, a significant jump over the 56.1% achieved by the baseline DFormer-L.
- Visual Quality: Qualitative results show the model is much better at capturing thin structures, such as the antennae and legs of a camouflaged shrimp, which are usually lost in standard depth-fusion models.
Limitations and Future Work
- Computational Overhead: While the authors mention the small spatial dimensions make it efficient, the iterative nature of the diffusion process and the use of FFTs add computational complexity compared to simple concatenation.
- Saturation: Performance gains tend to plateau after 4-5 diffusion iterations, suggesting there is a limit to how much "texture" a depth map can usefully hold.
- Future Directions: The authors suggest that this "texture-diffusion" philosophy could be extended to other multi-modal tasks beyond just RGB-D, potentially helping in fields like medical imaging or infrared-visible light fusion.
Summary updated