Abstract
Composed Image Retrieval (CIR) is a task that retrieves images similar to a query, based on a provided textual modification. Current techniques rely on supervised learning for CIR models using labeled triplets of the <reference image, text, target image>. These specific triplets are not as commonly available as simple image-text pairs, limiting the widespread use of CIR and its scalability. On the other hand, zero-shot CIR can be relatively easily trained with image-caption pairs without considering the image-to-image relation, but this approach tends to yield lower accuracy. We propose a new semi-supervised CIR approach where we search for a reference and its related target images in auxiliary data and learn our large language model-based Visual Delta Generator (VDG) to generate text describing the visual difference (i.e., visual delta) between the two. VDG, equipped with fluent language knowledge and being model agnostic, can generate pseudo triplets to boost the performance of CIR models. Our approach significantly improves the existing supervised learning approaches and achieves state-of-the-art results on the CIR benchmarks.
Introduction
Composed Image Retrieval (CIR) is a sophisticated computer vision task where a user searches for a target image using a reference image and a text-based modification, known as a "visual delta." For instance, a user might provide an image of a black dog and the text "make it a white dog sitting on a rug." While powerful, the primary challenge in CIR is the scarcity of high-quality training data. Training effective models typically requires triplets consisting of a reference image, a target image, and the corresponding textual delta.
Currently, the field is split between supervised CIR, which relies on expensive human-annotated triplets, and zero-shot CIR, which uses abundant image-caption pairs but often lacks the precise image-to-image reasoning necessary for high accuracy. This paper introduces a semi-supervised approach that bridges this gap using a Large Multi-modal Model (LMM) called the Visual Delta Generator (VDG).
The Visual Delta Generator (VDG)
The core of the proposed framework is the VDG, designed to automatically generate textual descriptions of the differences between two images.
As shown in the architecture diagram above, the VDG leverages a Large Language Model (specifically LLaMA2-13B) integrated with a vision encoder. The connection is facilitated by a "Vision Projector" based on the Q-Former from InstructBLIP. The training of the VDG occurs in two stages:
- Alignment: The model is pre-trained on standard image-text pairs to align visual features with the LLM's linguistic space.
- Instruction Tuning: The model is fine-tuned to follow specific instructions, such as analyzing a reference and target image to describe the transformation required to turn one into the other.
Semi-Supervised Framework
The VDG enables a semi-supervised training pipeline by acting as an automated annotator. It scans auxiliary image databases to find pairs of similar images and generates "pseudo-triplets."
By combining a small set of human-labeled "Supervised Triplets" with a massive volume of VDG-generated "Pseudo Triplets," the system can train retrieval models that generalize far better than those trained on limited human data alone. To handle the potential noise in synthetic data, the authors employ a Supervised/Pseudo Separated Contrastive Loss. They also introduce a Target-Delta Matching (TDM) loss, which ensures the generated text delta aligns closely with the visual features of the target image.
Experimental Results
The methodology was rigorously tested on two primary benchmarks: CIRR (natural images) and FashionIQ (clothing-specific images). The results demonstrate that the VDG-based approach achieves state-of-the-art (SOTA) performance.
On the CIRR test set, the BLIP-based model achieved a Recall@1 of 50.96%, outperforming previous leading methods like CoVR and CASE. In the FashionIQ benchmark, the model showed significant improvements across all categories, particularly in "Toptee," where it surpassed previous supervised models by a substantial margin. These results highlight that domain-specific pseudo-labeling is often more effective and efficient than brute-force training on massive, uncurated web-scale datasets.
Key Insights and Conclusion
The success of the Visual Delta Generator underscores the potential of Large Multi-modal Models to serve as high-quality, model-agnostic annotators for complex visual tasks. By generating fluent and contextually accurate visual deltas, the VDG allows CIR models to learn fine-grained relationships between images that were previously difficult to capture without massive human intervention.
While the approach is computationally intensive due to the use of a 13B parameter LLM, this is a one-time cost during the data generation phase. Future work may involve exploring even larger LMMs, improving noise filtering for pseudo-triplets, and extending the VDG framework to video-based retrieval tasks. In summary, the VDG provides a scalable and highly effective solution to the data bottleneck in Composed Image Retrieval.
Summary updated