Data Roaming and Early Fusion for Composed Image Retrieval

Matan Levy, Rami Ben-Ari, Nir Darshan, Dani Lischinski

Abstract

Composed Image Retrieval (CoIR) involves searching for a target image based on a query image and a text-based modification. Despite its potential, CoIR has been limited by small-scale datasets and models that struggle to effectively fuse visual and textual modalities. We present a summary of the work by Levy et al., which introduces the LaSCo dataset—a large-scale, open-domain corpus generated through a "Data Roaming" technique—and CASE, a Cross-Attention driven Shift Encoder. CASE employs early fusion to significantly outperform existing baselines on benchmarks like FashionIQ and CIRR while maintaining high retrieval efficiency.

Introduction

Composed Image Retrieval (CoIR) represents a sophisticated "search by example plus feedback" paradigm. A typical query consists of an image (e.g., a dog) and a text prompt (e.g., "sitting on a yellow pillow"). The model must synthesize these inputs to retrieve a target image that matches the requested composition.

Historically, CoIR has faced three major challenges:

  1. Data Scarcity: High-quality, human-annotated triplets are expensive and difficult to scale.
  2. Domain Specificity: Most datasets are confined to narrow categories like fashion.
  3. Modality Redundancy: Many queries allow models to ignore the image entirely, relying solely on descriptive text to find the target.

Scaling Data with Data Roaming

To overcome the limitations of manual annotation, the authors introduced "Data Roaming." This technique repurposes the VQA2.0 dataset, which contains "complementary" image pairs—images that are visually similar but yield different answers to the same question.

Data Roaming leverages VQA pairs to generate compositional queries by transforming question-answer pairs into transition text.

By using GPT-3 to rephrase (Question, Answer) pairs into natural language "transition text," the researchers automatically generated triplets of (Query Image, Transition Text, Target Image). This approach produced LaSCo (Large Scale Composed Image Retrieval), a dataset containing 389K queries and 121K unique images, making it approximately ten times larger than previous natural-image benchmarks.

The CASE Architecture: Early Fusion and Cross-Attention

Most existing CoIR models utilize "late fusion," where image and text features are encoded independently and combined only at the final stage. Levy et al. argue that this prevents deep interaction between modalities.

Their proposed model, CASE (Cross-Attention driven Shift Encoder), adopts an early fusion strategy. It uses a Vision Transformer (ViT) to decompose the query image into patches. These patches are then fed into a BERT-based Shift Encoder, where cross-attention layers allow text tokens to interact directly with visual features. This enables the model to more precisely "shift" the image representation toward the target.

The CASE architecture utilizes early fusion via cross-attention and incorporates a reverse-query objective for robust training.

Furthermore, CASE incorporates a Reverse-Query Objective. During training, the model is tasked with finding the query image given the target image and the text. This bidirectional learning acts as a powerful regularizer and data augmentation technique.

Results and Discussion

CASE established a new state-of-the-art on multiple benchmarks. On the FashionIQ dataset, it improved Recall@10 and Recall@50 by 13.4 and 11.6 absolute points, respectively. Beyond accuracy, CASE is highly efficient; by utilizing a 256D latent space, it achieves a 2.5x speedup in retrieval time compared to previous leaders.

The authors also addressed the "Modality Redundancy Trap." Their analysis revealed that while older datasets like CIRR are highly redundant (allowing text-only models to perform well), the new LaSCo dataset is truly compositional, requiring models to integrate both visual and textual information to succeed.

Conclusion

The introduction of LaSCo and the CASE model represents a significant step forward for Composed Image Retrieval. By leveraging automated data generation through Data Roaming and deep multimodal integration via early fusion, this work provides a scalable path toward general-purpose image search and editing-based retrieval systems. Future work may involve scaling these techniques to even larger VQA sources to develop foundation models for image composition.

Summary updated

BibTeX
@inproceedings{levy2024data,
  author = {Matan Levy and Rami Ben-Ari and Nir Darshan and Dani Lischinski},
  title = {{Data Roaming and Early Fusion for Composed Image Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28081},
}
Figure