DualCIR: Enhancing Training-Free Composed Image Retrieval via Dual-Directional Descriptions

Jingjiao Zhao, Jiaju Li, Dongze Lian, Liguo Sun, Pin LvWACV 2025 · DOI

Introduction

Composed Image Retrieval (CIR) is a sophisticated computer vision task where a user seeks a target image based on a reference image and a natural language modification text (e.g., "change the dog to a cat"). Traditionally, solving this task required training complex fusion networks on expensive, manually annotated triplet datasets. Recently, the field has shifted toward "training-free" or zero-shot methods that leverage the power of Large Language Models (LLMs) and Vision-Language Models (VLMs) like CLIP. While these methods avoid the need for specialized training data, they often struggle with the nuances of complex search intents.

The Challenge of Semantic Leakage

A significant hurdle for existing training-free methods, such as CIReVL, is the phenomenon of "semantic leakage." These models typically use an LLM to generate a single textual description of the target image. However, VLMs like CLIP often fail to process complex sentences with multiple constraints, particularly negations. For example, a query like "two people sitting on a bench without any bikes" might inadvertently lead the model to retrieve images containing bikes because the word "bike" is present in the prompt, even if preceded by "without."

Figure 3: Comparison between CIReVL and DualCIR. CIReVL's single description leads to semantic leakage (retrieving bikes), while DualCIR's positive and negative descriptions correctly isolate the target attributes.

As shown in the comparison above, standard holistic descriptions often fail to capture the exclusion criteria effectively. This limitation necessitates a more robust way to communicate search intent to the retrieval model without resorting to computationally expensive re-ranking steps.

The DualCIR Framework

DualCIR addresses semantic leakage by introducing a "dual-directional" approach that explicitly defines both what a target image should contain and what it should exclude. The framework operates in four distinct stages:

  1. Reference Captioning: An off-the-shelf captioner (e.g., BLIP-2) generates a detailed description of the initial reference image.
  2. Dual-directional Generation: Using Chain-of-Thought prompting, an LLM (such as GPT-4 or Vicuna) decomposes the search intent into three distinct descriptions:
  3. Holistic ($t_e$): A comprehensive summary of the target.
  4. Positive ($t_p$): Specific attributes and conditions that must be present.
  5. Negative ($t_n$): Attributes from the reference image that must be excluded.
  6. Multi-view Scoring: The system calculates similarity scores between candidate images and all three descriptions using a VLM backbone.
  7. Weighted Fusion: The final retrieval score is a weighted sum where the negative description score is subtracted, effectively penalizing images that contain unwanted features.

By leveraging batching during the VLM encoding phase, DualCIR achieves this multi-view perspective with negligible computational overhead compared to methods that require separate Visual Question Answering (VQA) calls for re-ranking.

Experimental Evaluation

The authors evaluated DualCIR across three major benchmarks: CIRCO (real-world COCO images), CIRR (natural images from NLVR2), and Fashion-IQ (domain-specific fashion retrieval).

On the CIRCO dataset, DualCIR achieved a state-of-the-art zero-shot mAP@5 of 27.86%, outperforming the previous training-free leader, CIReVL (25.69%). Notably, DualCIR's performance is competitive with, and in some cases superior to, models that undergo extensive training on specialized image-text datasets. The framework also demonstrated strong cross-domain consistency, showing marked improvements in the fashion domain where fine-grained attribute matching is critical.

Conclusion

DualCIR represents a significant step forward for training-free image retrieval. By treating the LLM as a logical decomposer rather than a simple caption generator, the framework overcomes the inherent limitations of VLMs in handling complex negations. While the system remains dependent on the quality of the initial image captioner, its efficiency and state-of-the-art performance make it a highly viable solution for real-world, zero-shot retrieval applications. Future work will likely focus on integrating more robust multi-scale vision models to further refine the granularity of the retrieval process.

Summary updated

BibTeX
@inproceedings{zhao2025dualcir,
  author = {Jingjiao Zhao and Jiaju Li and Dongze Lian and Liguo Sun and Pin Lv},
  title = {{DualCIR: Enhancing Training-Free Composed Image Retrieval via Dual-Directional Descriptions}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00578},
}
Figure