TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

Zixu Li, Yupeng Hu, Zhiheng Fu, Zhiwei Chen, Yongqi Li, Liqiang NiearXiv 2026 · arXiv

TL;DR

Composed Image Retrieval (CIR) models typically struggle when users provide long, complex text prompts containing multiple image modification requests. This paper introduces TEMA, a novel architecture that uses learnable queries and an LLM-based parsing assistant during training to cleanly map multiple textual clauses to their corresponding visual entities. Alongside the model, the authors release M-FashionIQ and M-CIRR, two new instruction-rich datasets designed to push CIR research toward realistic, multi-modification scenarios.

What Problem Does This Paper Address?

Composed Image Retrieval (CIR) is a multimodal task where a user searches for an image using a combination of a reference image and a modification text (e.g., showing a picture of a black dress and typing "make it magenta with a frill and longer sleeves").

While current CIR models perform well on short, simple text prompts, real-world queries are rarely that basic. When faced with complex, multi-modification instructions, existing models suffer from two major flaws:

  1. Insufficient Entity Coverage: Models tend to focus only on the most visually salient change requested in the text, completely ignoring secondary modifications.
  2. Clause-Entity Misalignment: Models struggle with complex sentence structures, such as when multiple distinct clauses modify a single object, or when one clause applies to multiple objects simultaneously.

To bridge this gap, the authors introduce both complex evaluation benchmarks and a architecture capable of handling intricate, multi-entity modifications.

Methods

The authors propose TEMA (Text-oriented Entity Mapping Architecture), which introduces two core components to handle complex Multi-Modification Text (MMT):

  • MMT Parsing Assistant (PA): Operating only during training, this module uses an LLM (GPT-3.5-turbo) to summarize long, detailed modification texts into a concise sentence that retains all target entities. A secondary Consistency Detector checks the summary to prevent LLM hallucinations. This summary acts as a clean guide to keep the model focused on all required changes. Because it is disabled during inference, it introduces zero real-time operational delay.
  • MMT-oriented Entity Mapping (EM): To resolve clause-entity misalignment, this module introduces a set of learnable queries inside a Transformer block. Guided by the PA summary, these queries aggregate scattered textual clauses describing the same entity and align them with the corresponding visual regions in the reference image—all without needing explicit bounding-box annotations.

TEMA is optimized using a joint loss function: a standard batch-based classification loss, a summary-guided distillation loss (aligning text tokens with the PA summary), and an orthogonal regularization loss that forces the learnable queries to remain semantically independent.

Datasets

The paper evaluates TEMA on both traditional benchmarks and two newly constructed, instruction-rich datasets:

  • M-FashionIQ (New): A fashion-domain dataset containing 24,016 queries expanded into detailed, multi-modification instructions.
  • M-CIRR (New): An open-domain dataset containing 32,406 queries featuring complex multi-object and background modifications.
  • Dataset Construction: The new datasets were generated using Llama 3.2 to describe differences between reference and target images, filtered for hallucinations using GPT-4o, and manually verified by human annotators.
  • FashionIQ & CIRR: The original, simple-text versions of these datasets were used to test TEMA's backward compatibility.

Key Insights and Ideas

  • Detailed Text Mitigates False Negatives: A long-standing issue in CIR is "false negatives"—visually similar images that technically match a short query but are labeled as incorrect. The authors found that expanding short text prompts into highly detailed instructions actually alleviates this issue. Denser text constraints make the true target uniquely identifiable, turning ambiguous false negatives into clear true negatives.
  • LLMs as "Training Wheels": Leveraging powerful LLMs strictly during the training phase allows the model to inherit dense supervisory signals regarding entity coverage without carrying the computational weight of an LLM during deployment.
  • The Power of Orthogonal Regularization: Forcing the learnable query channels to be orthogonal prevents them from overlapping semantically. This architectural constraint ensures that different queries track distinct entities cleanly rather than collapsing onto the same prominent object.

Key Results

TEMA achieves state-of-the-art performance across both complex and traditional CIR tasks while demonstrating superior computational efficiency.

Quantitative Performance

  • Multi-Modification Scenarios: On the new M-FashionIQ dataset, TEMA achieved an average R@10 of 50.59% and R@50 of 72.09%, outperforming the strongest baselines by roughly 3-5%. On M-CIRR, it achieved a top average score of 75.76%.
  • Traditional Scenarios: When tested on the original, simple-text benchmarks, TEMA still set new state-of-the-art results (e.g., hitting 53.02% R@10 on the original FashionIQ), proving it doesn't sacrifice simple retrieval performance to handle complex queries.

Efficiency

Compared to sub-optimal baselines like Candidate, TEMA reduces computational complexity and speeds up execution:

  • FLOPs: Reduced from 5.79G to 3.68G.
  • Inference Time: Cut in half, from 16.7ms to 7.9ms per sample.

Limitations and Future Work

  • Single-Turn Retrieval Only: TEMA is designed for a single snapshot query. It does not currently support multi-turn interactive CIR, where a user might iteratively refine their search through an ongoing dialogue.
  • Increased Text Complexity: While detailed text helps specify exact targets, it inherently increases linguistic complexity. If a user's prompt is poorly written or logically incoherent, text understanding can become a bottleneck.
  • Training Overhead: Although the LLM-based Parsing Assistant is disabled during testing, it still introduces minor computational overhead during the training phase.

Summary updated

Abstract

Composed Image Retrieval (CIR) is an important image retrieval paradigm that enables users to retrieve a target image using a multimodal query that consists of a reference image and modification text. Although research on CIR has made significant progress, prevailing setups still rely simple modification texts that typically cover only a limited range of salient changes, which induces two limitations highly relevant to practical applications, namely Insufficient Entity Coverage and Clause-Entity Misalignment. In order to address these issues and bring CIR closer to real-world use, we construct two instruction-rich multi-modification datasets, M-FashionIQ and M-CIRR. In addition, we propose TEMA, the Text-oriented Entity Mapping Architecture, which is the first CIR framework designed for multi-modification while also accommodating simple modifications. Extensive experiments on four benchmark datasets demonstrate that TEMA's superiority in both original and multi-modification scenarios, while maintaining an optimal balance between retrieval accuracy and computational efficiency. Our codes and constructed multi-modification dataset (M-FashionIQ and M-CIRR) are available at https://github.com/lee-zixu/ACL26-TEMA/.
BibTeX
@misc{li2026tema,
  author = {Zixu Li and Yupeng Hu and Zhiheng Fu and Zhiwei Chen and Yongqi Li and Liqiang Nie},
  title = {{TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval}},
  year = {2026},
  eprint = {2604.21806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.21806},
}

Loading related content...

Figure