Abstract
Composed Image Retrieval (CIR) involves searching for a target image based on a reference image and a text instruction (delta caption). Current approaches are limited by the scarcity of labeled triplet data and the inherent representation gap between text queries and image databases. We present Generative Zero-Shot Composed Image Retrieval (CIG), a framework that utilizes latent diffusion models to "imagine" a pseudo-target image. By converting the multimodal query into a purely visual representation, CIG significantly improves retrieval performance in a zero-shot setting.
Introduction
The task of Composed Image Retrieval (CIR) is challenging because it requires understanding both visual context and linguistic instructions. For example, a user might provide an image of a brown dog and the instruction "change the color to white." Supervised models for this task require massive datasets of (reference, caption, target) triplets, which are difficult to collect. Furthermore, zero-shot methods often struggle with the "representation gap," where text-based queries fail to capture the fine-grained visual details necessary for accurate retrieval from an image-only database.
Methods
The proposed CIG framework addresses these issues by generating a visual representation of the query before retrieval. The process involves three main components:
- Textual Inversion: A reference image is mapped into a semantic word embedding, or "pseudo-token," allowing it to be processed alongside natural language.
- Pseudo-Target Generation: A pre-trained Latent Diffusion Model (LDM) is used to generate an image based on the pseudo-token and the delta caption. This model is trained on standard image-caption pairs, avoiding the need for CIR triplets.
- Hybrid Retrieval: The final retrieval is performed by combining the features of the original query and the generated pseudo-target image to search the database.
Key Results
CIG was evaluated on several benchmarks, including CIRR, CIRCO, and Fashion-IQ. The method consistently improved performance over strong baselines. For instance, on the CIRR dataset, CIG boosted Top-1 Recall from 24.24% to 26.72% when added to the SEARLE model. By utilizing SDXL-Turbo, the system maintains high efficiency, adding only ~100ms of latency per query.
Qualitatively, the model demonstrates a remarkable ability to preserve the identity of the reference object while accurately applying complex textual modifications. As shown in the examples below, the diffusion model can handle changes in orientation, quantity, and specific attributes like food toppings.
Conclusion
Generative Zero-Shot Composed Image Retrieval (CIG) provides a scalable, "plug-and-play" solution for CIR. By leveraging the generative power of diffusion models, it bridges the gap between text and vision, achieving state-of-the-art zero-shot performance. Future work will focus on reducing latency further and improving attribute preservation in specialized domains like fashion.
Summary updated