Data Roaming and Quality Assessment for Composed Image Retrieval

Matan Levy, Rami Ben-Ari, Nir Darshan, Dani Lischinski

Abstract

Composed Image Retrieval (CoIR) requires models to retrieve a target image based on a multi-modal query consisting of a reference image and a modifying text. This task is currently hindered by data scarcity, modality redundancy (where text queries overshadow visual context), and evaluation noise. This paper introduces LaSCo, a large-scale dataset generated via an automated "Data Roaming" pipeline, and CASE, a novel early-fusion architecture. CASE utilizes a cross-attention-driven shift encoder and a bi-directional training objective to achieve state-of-the-art performance across major benchmarks while improving retrieval efficiency.

Introduction

Composed Image Retrieval (CoIR) represents a sophisticated search paradigm where users refine visual queries using natural language. For instance, a user might provide an image of a yellow dress and specify, "change the color to blue and add sleeves." Despite its potential, the field faces three critical challenges: the extreme scarcity of high-quality annotated data, the prevalence of "modality redundancy" (where models ignore the image because the text is overly descriptive), and the presence of unlabelled valid targets (false negatives) in existing benchmarks.

The LaSCo Dataset and Data Roaming

To address the data bottleneck, we introduce LaSCo (Large Scale Composed Image Retrieval). Rather than relying on expensive human annotation, we propose a "Data Roaming" pipeline that repurposes the VQA 2.0 dataset. By identifying image pairs that yield different answers to the same question, we isolate specific visual differences. We then employ GPT-3 to synthesize transition text that describes the shift from the source to the target image.

LaSCo comprises 389,000 queries and 121,000 open-domain images, making it ten times larger than previous datasets. Crucially, LaSCo is designed to minimize modality redundancy, ensuring that the model must integrate both visual and textual information to succeed. Human evaluations indicate that 90.9% of LaSCo’s automated annotations are high-quality, nearly matching the 93.8% rating of the human-annotated FashionIQ dataset.

The CASE Architecture

We propose CASE (Cross-Attention driven Shift Encoder) as a robust baseline for CoIR. Unlike traditional "late-fusion" models that encode modalities separately, CASE implements an early-fusion strategy. Built upon the BLIP foundation model, CASE allows text tokens and image patches to interact deeply through cross-attention layers within a dedicated Shift Encoder.

The CASE architecture features a ViT encoder and a Shift Encoder that utilizes cross-attention to fuse image and text features, supported by a Reverse-Query training objective.

Reverse-Query Objective

A key innovation in CASE is the Reverse-Query (RQ) objective. During training, the model is tasked with two simultaneous goals: finding the target image given the query (forward task) and predicting the original query image given the target image and the text (reverse task). This symmetric objective acts as a powerful regularizer, forcing the model to learn a more robust and reversible joint embedding space.

Experimental Results

CASE demonstrates significant performance gains over existing state-of-the-art (SOTA) methods:

  • Benchmark Dominance: On FashionIQ, CASE achieved a 48.48% Recall@10, a substantial improvement over the previous SOTA (LF-CLIP at 36.36%). On the CIRR dataset, it reached 48.00% Recall@1, compared to the previous best of 33.59%.
  • Zero-Shot Capability: Pre-training on LaSCo allows CASE to perform exceptionally well on unseen datasets. In a zero-shot setting on CIRR, CASE achieved 35.40% R@1, outperforming many models specifically trained on that data.
  • Efficiency: By projecting multi-modal queries into a compressed 256-dimensional space, CASE accelerates retrieval search speeds by 2.5x compared to higher-dimensional models (e.g., 640D).

Conclusion

By introducing the LaSCo dataset and the CASE architecture, this work provides a scalable solution to the core problems of CoIR. The automated Data Roaming pipeline demonstrates that LLMs can effectively generate high-quality multi-modal training data, while the CASE model sets a new standard for compositionality and retrieval efficiency. Future work will explore the utility of LaSCo for broader vision-language pre-training tasks.

Summary updated

Abstract

The task of Composed Image Retrieval (CoIR) involves queries that combine image and text modalities, allowing users to express their intent more effectively. However, current CoIR datasets are orders of magnitude smaller compared to other vision and language (V&L) datasets. Additionally, some of these datasets have noticeable issues, such as queries containing redundant modalities. To address these shortcomings, we introduce the Large Scale Composed Image Retrieval (LaSCo) dataset, a new CoIR dataset which is ten times larger than existing ones. Pre-training on our LaSCo, shows a noteworthy improvement in performance, even in zero-shot. Furthermore, we propose a new approach for analyzing CoIR datasets and methods, which detects modality redundancy or necessity, in queries. We also introduce a new CoIR baseline, the Cross-Attention driven Shift Encoder (CASE). This baseline allows for early fusion of modalities using a cross-attention module and employs an additional auxiliary task during training. Our experiments demonstrate that this new baseline outperforms the current state-of-the-art methods on established benchmarks like FashionIQ and CIRR.
BibTeX
@inproceedings{levy2024data,
  author = {Matan Levy and Rami Ben-Ari and Nir Darshan and Dani Lischinski},
  title = {{Data Roaming and Quality Assessment for Composed Image Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28081},
}
Figure