SceneDiff: Generative Scene-Level Image Retrieval with Text and Sketch Using Diffusion Models

Ran Zuo, Haoxiang Hu, Xiaoming Deng, Cangjun Gao, Zhengming Zhang, Yu-Kun Lai, Cuixia Ma, Yong-Jin Liu, Hongan WangIJCAI 2024

Introduction

Scene-level Text and Sketch-Based Image Retrieval (Scene-level TSBIR) is a sophisticated computer vision task where users provide a combination of a rough sketch and a textual description to locate a specific image. While text is effective for conveying semantic details like color and category, sketches provide indispensable structural information regarding layout and shape. SceneDiff introduces a novel framework that treats this retrieval task as a generative alignment problem, leveraging the latent space of pre-trained diffusion models to achieve superior accuracy and robustness.

Rethinking Multi-Modal Fusion

The primary challenge in TSBIR is the "complementary" nature of the inputs. Traditional methods often rely on "direct fusion" strategies, such as concatenating feature vectors or employing simple cross-attention mechanisms. However, these shallow fusion techniques frequently fail to capture the intricate relationships between modalities, leading to a loss of fine-grained structural detail.

Conceptual comparison between direct fusion and the proposed diffusion-based shared space.

As illustrated above, SceneDiff shifts away from direct fusion toward a shared generative latent space. By utilizing a pre-trained diffusion model (specifically Stable Diffusion), the framework maps disparate query modalities into a common domain where they can be more effectively aligned with target images.

The SceneDiff Architecture

The SceneDiff framework is built upon a multi-stage pipeline designed to transform and enhance multi-modal features within a generative context.

The complete SceneDiff architecture, including Content-aware Feature Transformation (CAFT) and Generated Feature Enhancement (GFEM).

1. Dual-Path Encoding and CAFT

The system begins by extracting initial features using CLIP encoders. Because standard diffusion models require pixel-level inputs while CLIP provides token-level representations, the Content-aware Feature Transformation (CAFT) module is employed. CAFT uses Depthwise Separable Convolutions (DWConv) and fully connected layers to reshape these features, ensuring they match the diffusion model's requirements while preserving visual fidelity.

2. Generative Latent Fusion

The transformed sketch and text features serve as conditions for a denoising process. By mapping these queries into the image domain's latent space, the model produces "generated latent fusion features." A key insight from the authors is that full image generation is unnecessary; sampling at an early denoising step ($k=2$) provides sufficient semantic and structural information for retrieval, significantly reducing computational overhead.

3. Generated Feature Enhancement Module (GFEM)

To mitigate the stochastic nature of diffusion, the GFEM integrates multiple samples from intermediate denoising steps. It utilizes a "partition attention" mechanism, applying cross-attention to localized regions to synthesize a robust and representative feature vector.

4. Two-Level Contrastive Learning

Training is guided by a dual alignment strategy: aligning directly fused CLIP features with the target image, and aligning the generated latent fusion features with the latent representation produced by the diffusion model's autoencoder.

Experimental Evaluation

SceneDiff was evaluated against state-of-the-art (SOTA) models like TASK-former and SceneTrilogy across three major benchmarks:

  • SFSD: Achieved a Recall@1 (R@1) of 85.1%, surpassing the previous SOTA of 78.52%.
  • FS-COCO: Reached 46.37% R@1, a significant improvement over the 40.27% benchmark.
  • SketchyCOCO: Reached 61.02% R@1, nearly doubling the performance of several baseline methods.

Ablation studies highlighted the critical role of the sketch as a structural "anchor." While the model remains robust to missing text, performance drops sharply without sketch information, confirming that sketches provide the essential layout backbone for scene-level queries.

Conclusion

SceneDiff demonstrates that the latent space of diffusion models can serve as a powerful bridge for multi-modal alignment. By reframing retrieval as a generative process, the framework achieves unprecedented accuracy and maintains high performance even with incomplete or sparse user queries. Future work may extend this generative approach to more complex domains, such as video retrieval or 3D scene synthesis.

Summary updated

BibTeX
@inproceedings{zuo2024scenediff,
  author = {Ran Zuo and Haoxiang Hu and Xiaoming Deng and Cangjun Gao and Zhengming Zhang and Yu-Kun Lai and Cuixia Ma and Yong-Jin Liu and Hongan Wang},
  title = {{SceneDiff: Generative Scene-Level Image Retrieval with Text and Sketch Using Diffusion Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure