Abstract
Composed Image Retrieval (CoIR) requires models to retrieve a target image based on a multi-modal query consisting of a reference image and a modifying text. This task is currently hindered by data scarcity, modality redundancy (where text queries overshadow visual context), and evaluation noise. This paper introduces LaSCo, a large-scale dataset generated via an automated "Data Roaming" pipeline, and CASE, a novel early-fusion architecture. CASE utilizes a cross-attention-driven shift encoder and a bi-directional training objective to achieve state-of-the-art performance across major benchmarks while improving retrieval efficiency.
Introduction
Composed Image Retrieval (CoIR) represents a sophisticated search paradigm where users refine visual queries using natural language. For instance, a user might provide an image of a yellow dress and specify, "change the color to blue and add sleeves." Despite its potential, the field faces three critical challenges: the extreme scarcity of high-quality annotated data, the prevalence of "modality redundancy" (where models ignore the image because the text is overly descriptive), and the presence of unlabelled valid targets (false negatives) in existing benchmarks.
The LaSCo Dataset and Data Roaming
To address the data bottleneck, we introduce LaSCo (Large Scale Composed Image Retrieval). Rather than relying on expensive human annotation, we propose a "Data Roaming" pipeline that repurposes the VQA 2.0 dataset. By identifying image pairs that yield different answers to the same question, we isolate specific visual differences. We then employ GPT-3 to synthesize transition text that describes the shift from the source to the target image.
LaSCo comprises 389,000 queries and 121,000 open-domain images, making it ten times larger than previous datasets. Crucially, LaSCo is designed to minimize modality redundancy, ensuring that the model must integrate both visual and textual information to succeed. Human evaluations indicate that 90.9% of LaSCo’s automated annotations are high-quality, nearly matching the 93.8% rating of the human-annotated FashionIQ dataset.
The CASE Architecture
We propose CASE (Cross-Attention driven Shift Encoder) as a robust baseline for CoIR. Unlike traditional "late-fusion" models that encode modalities separately, CASE implements an early-fusion strategy. Built upon the BLIP foundation model, CASE allows text tokens and image patches to interact deeply through cross-attention layers within a dedicated Shift Encoder.
Reverse-Query Objective
A key innovation in CASE is the Reverse-Query (RQ) objective. During training, the model is tasked with two simultaneous goals: finding the target image given the query (forward task) and predicting the original query image given the target image and the text (reverse task). This symmetric objective acts as a powerful regularizer, forcing the model to learn a more robust and reversible joint embedding space.
Experimental Results
CASE demonstrates significant performance gains over existing state-of-the-art (SOTA) methods:
- Benchmark Dominance: On FashionIQ, CASE achieved a 48.48% Recall@10, a substantial improvement over the previous SOTA (LF-CLIP at 36.36%). On the CIRR dataset, it reached 48.00% Recall@1, compared to the previous best of 33.59%.
- Zero-Shot Capability: Pre-training on LaSCo allows CASE to perform exceptionally well on unseen datasets. In a zero-shot setting on CIRR, CASE achieved 35.40% R@1, outperforming many models specifically trained on that data.
- Efficiency: By projecting multi-modal queries into a compressed 256-dimensional space, CASE accelerates retrieval search speeds by 2.5x compared to higher-dimensional models (e.g., 640D).
Conclusion
By introducing the LaSCo dataset and the CASE architecture, this work provides a scalable solution to the core problems of CoIR. The automated Data Roaming pipeline demonstrates that LLMs can effectively generate high-quality multi-modal training data, while the CASE model sets a new standard for compositionality and retrieval efficiency. Future work will explore the utility of LaSCo for broader vision-language pre-training tasks.
Summary updated