Zero-Shot Composed Image Retrieval Considering Query-Target Relationship Leveraging Masked Image-Text Pairs

Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama

Abstract

Composed Image Retrieval (CIR) aims to retrieve a target image based on a query consisting of both an image and a modifying text. While supervised methods achieve high performance, they rely on labor-intensive human-annotated triplets. This paper presents a novel zero-shot CIR framework that simulates the query-target relationship using standard image-text pairs. By employing a masking strategy on both text and images, the proposed method (PM) trains a lightweight network to reconstruct a target image from a partial image and a descriptive text. Experimental results on CIRR and FashionIQ benchmarks demonstrate that this approach significantly outperforms existing zero-shot benchmarks and narrows the gap with supervised models.

Introduction

Composed Image Retrieval (CIR) is a challenging computer vision task where a user seeks a specific target image by providing a reference image and a text describing desired modifications (e.g., an image of a dog plus the text "running on a beach"). Traditional supervised approaches require massive datasets of $(I_{query}, T_{query}, I_{target})$ triplets. Collecting these triplets is exceptionally difficult and costly, limiting the scalability of supervised CIR.

To address this, researchers have turned to Zero-Shot CIR. Most zero-shot methods utilize pre-trained Vision-Language Models (VLMs) like CLIP and a "textual inversion" network. This network converts the query image into a "pseudo-word" that can be concatenated with the query text to form a unified prompt. However, a fundamental flaw exists in current zero-shot paradigms: the inversion network is typically trained to match a pseudo-word back to its original source image. This ignores the actual retrieval relationship—how the query image and text should interact to find a different target image—creating a significant discrepancy between training objectives and inference tasks.

Methodology

The authors propose a method, referred to as PM, which bridges the training-inference gap by synthesizing retrieval triplets from standard image-text pairs $(I, T)$. The core of the approach is a masking strategy that treats the original image as the "target" and a modified version as the "query."

1. Image-Text Masking

The method creates a synthetic query through two synchronized masking steps:

  • Text Masking: The system identifies a noun within the caption $T$ and removes it to create a masked text $T^M$.
  • Image Masking: Using Class Activation Maps (CAM), the model identifies the visual region in image $I$ corresponding to the removed noun. This region is masked and replaced with patches from a different image in the training batch, creating a masked image $I^M$. This replacement with external patches simulates real-world noise and background complexity better than simple black-box masking.

2. Query Composition and Training

A lightweight Multi-Layer Perceptron (MLP) serves as the textual inversion network, converting the masked image $I^M$ into a pseudo-word $w^M$. This word is inserted into the masked text $T^M$ to form a "composed query" $T^C$.

The model is trained end-to-end using two primary loss functions:

  • Query-Target Loss ($L_{qt}$): This forces the embedding of the composed query $T^C$ to align with the original, unmasked image $I$ in the CLIP embedding space, effectively simulating the CIR task.
  • Original Loss ($L_{org}$): A standard reconstruction loss ensuring the inversion network maintains the ability to represent original image features accurately.

Experimental Results

Datasets and Setup

The model was trained on a subset of 250,000 image-text pairs from the Conceptual Captions (CC3M) dataset. Evaluation was conducted on two major benchmarks:

  • CIRR (Composed Image Retrieval on Real-life images): For general-purpose retrieval.
  • FashionIQ: For domain-specific retrieval (dresses, shirts, and toptees).

Performance

The proposed method achieved state-of-the-art results for zero-shot CIR:

  • CIRR Benchmark: PM achieved a Recall@1 of 26.1%, surpassing previous leaders like SEARLE-XL (24.2%) and the Pic2Word baseline (23.9%). Notably, this performance significantly narrows the gap with fully supervised methods, which typically reach approximately 30.3% R@1.
  • FashionIQ Benchmark: PM achieved the highest average Recall@10 (25.8%) among zero-shot competitors, demonstrating robust domain generalization beyond the general images found in CC3M.

Discussion and Insights

The success of PM stems from its ability to synthesize meaningful triplets without human intervention. By masking a specific object and its corresponding word, the model learns the natural relationship between a context (the query) and a whole (the target). Furthermore, the retrieval-focused inversion ensures that the network is optimized for the specific task of finding a target, rather than just reconstructing an input.

Limitations

Despite its success, the authors identified several areas for improvement:

  • Textual Bias: In some failure cases, the model over-relies on the text query. For instance, if the text specifies "blue water," the model may retrieve any image containing blue water while ignoring the subject of the query image.
  • Masking Complexity: The current iteration only masks a single noun per pair. More sophisticated strategies involving multiple elements or diverse captioning could further enhance performance.
  • CAM Dependency: The accuracy of the image masking is tied to the precision of the Class Activation Maps, which can occasionally be noisy.

Conclusion

This research demonstrates that zero-shot CIR can be significantly improved by aligning the training objective with the retrieval task. By leveraging masked image-text pairs to simulate query-target relationships, the proposed method sets a new state-of-the-art for zero-shot retrieval, providing a scalable alternative to supervised learning in complex vision-language tasks.

Summary updated

BibTeX
@inproceedings{zhang2024zero,
  author = {Huaying Zhang and Rintaro Yanagi and Ren Togo and Takahiro Ogawa and Miki Haseyama},
  title = {{Zero-Shot Composed Image Retrieval Considering Query-Target Relationship Leveraging Masked Image-Text Pairs}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10648023},
}
Figure