good4cir: Generating Detailed Synthetic Captions for Composed Image Retrieval

Pranavi Kolouju, Eric Xing, Robert Pless, Nathan Jacobs, Abby Stylianou

Abstract

Composed Image Retrieval (CIR) is a multi-modal search task where a user provides a reference image and a textual modification to find a specific target image. Despite recent progress in vision-language models (VLMs), the field remains constrained by the scarcity and quality of annotated datasets. Existing datasets often suffer from simplistic annotations or high manual labeling costs. We introduce good4cir, a structured pipeline that leverages VLMs to generate high-quality, fine-grained synthetic captions. By decomposing the captioning process into object-level descriptions and consistency checks, our method reduces hallucinations and enhances modification diversity. We demonstrate the effectiveness of our approach by improving existing benchmarks ($CIRR_R$) and introducing a challenging new domain (Hotel-CIR). Models trained on our synthetic data show significant improvements in retrieval accuracy and zero-shot generalization.

Introduction

In Composed Image Retrieval, the goal is to retrieve a target image based on a query consisting of a reference image and a text prompt describing the desired changes. This task requires a deep understanding of both visual features and linguistic nuances. However, the development of robust CIR models is hindered by data limitations. Human-annotated datasets like CIRR are difficult to scale, while automated mining often produces "lazy" captions that describe only the most obvious changes, ignoring fine-grained details.

Furthermore, when large vision-language models (VLMs) are used to generate differences between images in a single step, they frequently suffer from hallucinations—imagining objects not present in the scene or failing to maintain consistency across the image pair. To address these challenges, we propose good4cir, a four-stage pipeline designed to produce structured, detailed, and consistent synthetic captions.

The good4cir Pipeline

The core philosophy of good4cir is that structure beats scale. Rather than asking a VLM to describe the difference between two images in one go, we force the model to perform "bookkeeping" through a multi-stage process:

  1. Stage 1: Query Image Object Descriptions: The VLM identifies key objects in the reference image and generates detailed descriptors for each, including attributes like color, texture, and spatial position.
  2. Stage 2: Target Image Object Descriptions: The VLM analyzes the target image while referencing the list from Stage 1. It must explicitly categorize each object as new, identical, or modified. This ensures object-level consistency, preventing the model from overlooking background elements when foreground objects change.
  3. Stage 3: Describing Differences: Using the structured lists from the previous stages, the VLM synthesizes natural language instructions (e.g., "Replace the wooden chair with a metal one and add a blue cushion").
  4. Stage 4: Caption Permutations: To increase complexity, individual modifications are automatically combined into compound sentences. We use a CLIP-based tokenizer to ensure these complex instructions remain within the model's effective processing limits.

Datasets and Benchmarks

We applied the good4cir pipeline to create and refine several key benchmarks:

  • $CIRR_R$: A re-written version of the classic CIRR dataset. By replacing original, simplistic captions with richer, multi-faceted descriptions, we provide a more challenging training signal.
  • Hotel-CIR: A new, highly challenging dataset derived from the Hotels-50K collection. This domain is "object-dense," requiring models to distinguish between similar hotel rooms based on subtle changes in furniture, decor, or layout.
  • CIRCO: A zero-shot benchmark used to evaluate how well models trained on our synthetic data generalize to unseen tasks and domains.

Results and Analysis

Our experiments demonstrate that the structured granularity of good4cir significantly improves retrieval performance across various settings.

Performance Gains

On the original CIRR test set, a model trained on a combination of original and $CIRR_R$ data improved Recall@1 from 16.5% to 19.1%. This suggests that the detailed synthetic captions help the model learn more robust features than human labels alone.

Zero-Shot Generalization

Training on synthetic $CIRR_R$ and Hotel-CIR data nearly doubled the mAP score on the CIRCO benchmark, increasing from 2.54 to 4.64 at mAP@5. This indicates that the diversity and detail of our synthetic captions help models generalize better to diverse retrieval scenarios.

Domain Mastery

In the object-dense Hotel-CIR dataset, models trained on good4cir-generated data outperformed the baseline by a massive margin, achieving 12.35% vs 2.03% at R@10. This highlights the importance of fine-grained descriptions in domains where visual differences are subtle.

Discussion and Limitations

While good4cir represents a significant step forward, some challenges remain. Despite the structured pipeline, VLMs still occasionally exhibit residual hallucinations in very similar image pairs. Additionally, like many current AI models, the pipeline struggles with precise counting (e.g., misidentifying the number of bottles in a scene). Finally, the reliance on high-end VLMs like GPT-4o introduces an API cost, though it remains more scalable than human annotation. Future work will focus on improving counting accuracy and exploring the use of open-source VLM alternatives to reduce costs.

Conclusion

The good4cir framework demonstrates that structured, multi-stage prompting can overcome the limitations of traditional synthetic data generation for CIR. By focusing on object-level consistency and fine-grained detail, we provide a scalable way to train more capable and generalized retrieval models. We release our dataset construction framework to support further research in multi-modal retrieval.

Summary updated

Abstract

Composed image retrieval (CIR) enables users to search images using a reference image combined with textual modifications. Recent advances in vision-language models have improved CIR, but dataset limitations remain a barrier. Existing datasets often rely on simplistic, ambiguous, or insufficient manual annotations, hindering fine-grained retrieval. We introduce good4cir, a structured pipeline leveraging vision-language models to generate high-quality synthetic annotations. Our method involves: (1) extracting fine-grained object descriptions from query images, (2) generating comparable descriptions for target images, and (3) synthesizing textual instructions capturing meaningful transformations between images. This reduces hallucination, enhances modification diversity, and ensures object-level consistency. Applying our method improves existing datasets and enables creating new datasets across diverse domains. Results demonstrate improved retrieval accuracy for CIR models trained on our pipeline-generated datasets. We release our dataset construction framework to support further research in CIR and multi-modal retrieval.
BibTeX
@inproceedings{kolouju2025good,
  author = {Pranavi Kolouju and Eric Xing and Robert Pless and Nathan Jacobs and Abby Stylianou},
  title = {{good4cir: Generating Detailed Synthetic Captions for Composed Image Retrieval}},
  booktitle = {Workshops @ CVPR},
  year = {2025},
}
Figure