Introduction
Composed Image Retrieval (CIR) is a sophisticated computer vision task where a user seeks a target image based on a reference image and a natural language modification text (e.g., "change the dog to a cat"). Traditionally, solving this task required training complex fusion networks on expensive, manually annotated triplet datasets. Recently, the field has shifted toward "training-free" or zero-shot methods that leverage the power of Large Language Models (LLMs) and Vision-Language Models (VLMs) like CLIP. While these methods avoid the need for specialized training data, they often struggle with the nuances of complex search intents.
The Challenge of Semantic Leakage
A significant hurdle for existing training-free methods, such as CIReVL, is the phenomenon of "semantic leakage." These models typically use an LLM to generate a single textual description of the target image. However, VLMs like CLIP often fail to process complex sentences with multiple constraints, particularly negations. For example, a query like "two people sitting on a bench without any bikes" might inadvertently lead the model to retrieve images containing bikes because the word "bike" is present in the prompt, even if preceded by "without."
As shown in the comparison above, standard holistic descriptions often fail to capture the exclusion criteria effectively. This limitation necessitates a more robust way to communicate search intent to the retrieval model without resorting to computationally expensive re-ranking steps.
The DualCIR Framework
DualCIR addresses semantic leakage by introducing a "dual-directional" approach that explicitly defines both what a target image should contain and what it should exclude. The framework operates in four distinct stages:
- Reference Captioning: An off-the-shelf captioner (e.g., BLIP-2) generates a detailed description of the initial reference image.
- Dual-directional Generation: Using Chain-of-Thought prompting, an LLM (such as GPT-4 or Vicuna) decomposes the search intent into three distinct descriptions:
- Holistic ($t_e$): A comprehensive summary of the target.
- Positive ($t_p$): Specific attributes and conditions that must be present.
- Negative ($t_n$): Attributes from the reference image that must be excluded.
- Multi-view Scoring: The system calculates similarity scores between candidate images and all three descriptions using a VLM backbone.
- Weighted Fusion: The final retrieval score is a weighted sum where the negative description score is subtracted, effectively penalizing images that contain unwanted features.
By leveraging batching during the VLM encoding phase, DualCIR achieves this multi-view perspective with negligible computational overhead compared to methods that require separate Visual Question Answering (VQA) calls for re-ranking.
Experimental Evaluation
The authors evaluated DualCIR across three major benchmarks: CIRCO (real-world COCO images), CIRR (natural images from NLVR2), and Fashion-IQ (domain-specific fashion retrieval).
On the CIRCO dataset, DualCIR achieved a state-of-the-art zero-shot mAP@5 of 27.86%, outperforming the previous training-free leader, CIReVL (25.69%). Notably, DualCIR's performance is competitive with, and in some cases superior to, models that undergo extensive training on specialized image-text datasets. The framework also demonstrated strong cross-domain consistency, showing marked improvements in the fashion domain where fine-grained attribute matching is critical.
Conclusion
DualCIR represents a significant step forward for training-free image retrieval. By treating the LLM as a logical decomposer rather than a simple caption generator, the framework overcomes the inherent limitations of VLMs in handling complex negations. While the system remains dependent on the quality of the initial image captioner, its efficiency and state-of-the-art performance make it a highly viable solution for real-world, zero-shot retrieval applications. Future work will likely focus on integrating more robust multi-scale vision models to further refine the granularity of the retrieval process.
Summary updated