MulVec: Fine-Grained Role-Aware Matching for Training-Free Zero-Shot Composed Image Retrieval

Zihao Zhang, Dayan Wu, Xinze Liu, Hengjie Zhu, Yiliang Zhu, Ding Wang, Peng Fu, Zheng Lin, Weiping WangarXiv 2026 · arXiv

TL;DR

TAPR introduces Atomic Edit Factorization to break down complex, entangled text edit requests into four distinct semantic roles: Target, Add, Preserve, and Remove. By evaluating these atomic constraints independently against local and global image features using factor-specific operators, TAPR achieves massive performance gains in training-free zero-shot composed image retrieval without requiring any task-specific fine-tuning.

What Problem Does This Paper Address?

Zero-Shot Composed Image Retrieval (ZS-CIR) is the task of searching an image gallery using a reference image combined with a natural language text edit (e.g., "Make the dog older and add two birds next to him"). Because it is "zero-shot" and "training-free," methods must rely entirely on pre-trained, frozen large language models (LLMs) and vision-language models (VLMs).

Existing state-of-the-art methods typically handle this by generating a single, precise description of the final target image. However, they then compress this description into a single global text vector and match it against a global image vector.

This creates a major bottleneck: semantic entanglement. When everything is compressed into a single global similarity score, dominant visual features (like a large dog) can easily overwhelm and mask fine-grained requirements (like two small birds or a specific painting style). Furthermore, global matching cannot inherently distinguish between what content should be newly added, what must be preserved from the original reference image, and what needs to be removed.

Fig. 1. Motivation on CIRR validation query 37730. Left: CoTMR’s global description matching [13] encodes one precise target description and compares it with one global visual vector per candidate, but its retrieval misses the requested birds and painting style. Right: TAPR factorizes the request into Target, Add, Preserve, and Remove factors, splits their constraints into atoms, and checks factor-specific probes against global and local candidate evidence to retrieve the ground-truth target at rank one.

Methods

The authors propose TAPR, a training-free framework that treats an edit request as a structured set of independent constraints rather than a single holistic description. The pipeline operates in three main stages:

  1. Atomic Edit Factorization: A frozen LLM (Qwen3.6-27B) acts as a "factorizer," parsing the text edit and reference context into four distinct semantic factors:
  2. Target ($T$): Describes the complete, holistic desired scene.
  3. Add ($A$): Lists specific visual elements that must appear.
  4. Preserve ($P$): Identifies source identities or contexts that must remain.
  5. Remove ($R$): Highlights states or objects that must disappear.
  6. Fine-Grained Multi-Vector Matching: Instead of relying solely on global image embeddings, TAPR constructs a Candidate Evidence Bank for each gallery image using a frozen VLM (OpenCLIP). This bank consists of one global visual vector and a set of local visual vectors (patch tokens).
  7. Factor-Specific Operators: Each factor is evaluated using a customized mathematical operator via a smooth late-interaction mechanism:
  8. Target performs standard global matching.
  9. Add rewards candidates that show local evidence of the new attributes.
  10. Remove applies a negative weight to penalize candidates containing forbidden attributes.
  11. Preserve enforces continuity by checking local candidate features directly against the reference image's local features.

The final retrieval score is a clean, linear combination of these factorized components: $$S = \theta_T \Phi_T + \theta_A \Phi_A + \theta_P \Phi_P - \theta_R \Phi_R$$

The TAPR Pipeline Overview

Datasets

The framework was evaluated entirely at inference time across three widely recognized CIR benchmarks:

  • CIRCO: An open-domain dataset focusing on multiple ground-truth targets per query, evaluated using mAP@k.
  • CIRR: A real-life image dataset evaluated using full-gallery Recall@k.
  • FashionIQ: A domain-specific dataset focused on detailed clothing attribute modifications (Shirts, Dresses, Toptees).

Key Insights and Ideas

  • Asymmetry is Essential for Frozen Models: A frozen model cannot implicitly learn what "Remove" means through standard cosine similarity. Explicitly assigning a negative sign to the Remove factor acts as an essential mathematical anchor that prevents false positives.
  • Local Patches Prevent Feature Masking: By evaluating Add and Remove constraints at the local patch level, small but critical details (like an Arabic script on a stop sign) are preserved and verified independently of the dominant background layout.
  • Independence Over Fusion: TAPR avoids complex multi-stage rank fusion or generating proxy images. It scores the real gallery in a single, direct pass, proving that structured query formulation is highly efficient.

Key Results

TAPR achieved state-of-the-art performance across all three datasets and across multiple backbone scales (ViT-B/32, ViT-L/14, and ViT-G/14):

  • CIRCO Benchmark: TAPR outperformed the strongest baseline (SDR-CIR) by a massive margin, improving mAP@5 by 23.0% on the ViT-B backbone and 22.4% on ViT-L.
  • CIRR & FashionIQ: TAPR consistently captured the top spot across all standard evaluation metrics, demonstrating that factorized matching works equally well for open-domain scene changes and fine-grained fashion attribute editing.

Limitations and Future Work

While TAPR significantly improves how text constraints are matched, it remains bounded by the capabilities of its underlying frozen components:

  • Encoder Blind Spots: If the pre-trained VLM fails to encode a specific fine-grained attribute or complex spatial relationship (such as drastic camera viewpoint changes), TAPR cannot magically surface it.
  • Factorization Errors: The system relies heavily on the LLM's ability to accurately split the prompt. Highly ambiguous text edits can lead to imperfect factor assignments.

The authors suggest that future work will focus on improving the extraction of edit atoms and developing more advanced region-aware evidence modeling while strictly maintaining the training-free nature of the system.

Summary updated

Abstract

Training-free zero-shot composed image retrieval finds a target image in a gallery from a reference image and a text edit without learning from task-specific image triplets. Existing methods typically describe the target as a whole and match this description with a global image representation. This global matching can mix different semantic cues and lose fine- grained details. We propose MULVEC, a role-aware method whose compiler produces a structured query record that is mapped to four retrieval roles: Global describes the full target, Desired states what should appear, Preserve states what should remain, and Forbidden states what should disappear. Frozen encoders map the query to one target description vector and role-specific probe vectors, while each candidate is represented by one global visual vector and a bank of local visual vectors. The retrieval roles then use this shared evidence for their respective purposes, and a fixed weighted sum of their scores ranks the entire gallery in a single retrieval pass. Across CIRCO, CIRR, and FashionIQ and three backbone scales, MULVEC improves CIRCO mAP@5 by up to 23.0% over the strongest compared method and gives the best CIRR and FashionIQ results in our comparison.
BibTeX
@misc{zhang2026mulvec,
  author = {Zihao Zhang and Dayan Wu and Xinze Liu and Hengjie Zhu and Yiliang Zhu and Ding Wang and Peng Fu and Zheng Lin and Weiping Wang},
  title = {{MulVec: Fine-Grained Role-Aware Matching for Training-Free Zero-Shot Composed Image Retrieval}},
  year = {2026},
  eprint = {2608.25305},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.25305},
}
Figure