MIPR: A Benchmark for Reasoning-Intensive Multimodal Intent-to-Product Retrieval with Multi-Category Solutions

Qiang Li, Xuan Lu, Jian Ruan, Manxi Lin, Jiacheng Chen, Wen Shen, Zhihua Wei, Xiaoyu Shen, Dan Ou, Xiangheng Kong, Junjun ZhengarXiv 2026 · arXiv

In the world of e-commerce, we’ve gotten very good at "target-centric" search. If you type "blue Nike running shoes," modern algorithms will find them in milliseconds. But real-world shopping is often messier. Sometimes we have a problem, not a product name. We might snap a photo of a rusty kitchen knife and ask, "How can I make this look new again?"

A new paper introduces MIPR, a benchmark designed to test whether AI can handle these "intent-centric" queries that require deep reasoning and diverse solutions.

TL;DR

MIPR is a new reasoning-intensive benchmark for multimodal e-commerce retrieval that requires models to infer user intent from image-text pairs. Unlike traditional benchmarks, it features multi-category solutions and fine-grained relevance labels, revealing that even state-of-the-art models struggle to find diverse products that satisfy complex user needs.

What Problem Does This Paper Address?

Existing multimodal retrieval benchmarks typically assume a well-defined target. However, real-world user needs are often open-ended. A single intent—like restoring a rusty knife—can be fulfilled by products from entirely different categories: a chemical rust remover, a specialized rust eraser, or even high-grit sandpaper.

As illustrated in Figure 1: An example of the intent-to-product retrieval task., current models face three major hurdles:

  1. Reasoning-Intensive Retrieval: The model must infer the intent (rust removal) rather than just matching keywords.
  2. Multi-Category Solutions: A single query might have valid solutions across multiple functional categories.
  3. Fine-Grained Constraints: Models must distinguish between "optimal" solutions (food-safe removers) and "suboptimal" or "constraint-violating" ones (industrial-grade chemicals).
Figure 1: An example from MIPR. Each query combines an image and a question to express a user intent without specifying a target product, requiring models to understand the intent and reason about which products can fulfill it while satisfying any applicable constraints. Products are assigned fine-grained relevance labels, and the leaf-level categories of L1 products are defined as solution categories. Text is translated and lightly edited for clarity.

Methods

The authors developed a sophisticated pipeline to build MIPR from real Chinese e-commerce search logs (Taobao):

  • Query Curation: They filtered ~130k raw logs down to ~3.5k high-quality queries that require joint multimodal understanding and have objective solutions.
  • Candidate Discovery: Using LLMs, they generated diverse "product terms" (e.g., "baking soda," "rust eraser") to ensure the candidate pool covered various solution types.
  • Fine-Grained Annotation: They used a "council of LLMs" (including Qwen, GLM, and DeepSeek) with human adjudication to label products across five levels: L1 (Optimal), L2 (Suboptimal), N1 (Constraint-violating), N2 (Related non-solution), and N3 (Irrelevant).
  • Pooling: To ensure the benchmark is robust, they merged the top results from ten different state-of-the-art retrievers and annotated any previously unjudged pairs.
Figure 2: Overview of the MIPR construction pipeline. Starting from real multimodal search queries and a large product catalog, we curate eligible queries, discover candidate products through complementary retrieval routes, construct a deduplicated shared corpus, assign five-way relevance labels, and complete the judgments through retrieval-based pooling.

Datasets

The MIPR (Multimodal Intent-to-Product Retrieval) benchmark includes:

  • 3,522 multimodal queries (image + question).
  • 30,731 products in a shared corpus.
  • 447,809 fine-grained relevance judgments. The dataset is grounded in real-world search behavior and is publicly available on Hugging Face.

Key Insights and Ideas

  • The Power of Both Modalities: Modality ablation studies show that using both image and text is essential. As shown in Figure 3: Modality ablation and the impact of explicit intent., multimodal queries significantly outperform text-only or image-only versions. The image provides the situational context, while the text defines the specific need.
  • The Reasoning Gap: When the authors manually added the "inferred intent" to the query, performance jumped significantly. This suggests that the "bottleneck" for current models isn't just retrieval—it's the internal reasoning step required to bridge the gap from a problem to a solution.
  • Hard Negatives are Truly Hard: Models frequently rank "related non-solutions" (N2) highly. For example, if you ask how to fix a knife, a model might suggest buying a new knife. While semantically related, it fails to fulfill the user's specific intent of restoration.
Figure 3: nDCG@10 under modality ablation and query aug- mentation. (a) Performance using image-only, text-only, and multimodal queries. (b) Performance after augmenting mul- timodal queries with intent descriptions and generated prod- uct terms.

Key Results

  • Performance Ceiling: Even the best embedding-based retriever (Qwen3-VL-Embedding) only achieved an nDCG@10 of 49.23%.
  • Reranking is Better but Not Enough: Applying a specialized reranker improved the score to 60.11%, but a massive gap remains compared to the "oracle" upper bound (86.04%).
  • Category Coverage: Models struggle to provide a diverse set of solutions. For queries with multiple valid categories, the best model only recalled 55.86% of those categories in its top 10 results. This "solution-category recall" drops sharply as the number of possible categories increases, as seen in Figure 5: Performance vs. number of solution categories..
Figure 5: Performance vs. number of solution categories.

Limitations and Future Work

The authors acknowledge that while their pooling-based approach is thorough, it is inherently limited to the products discovered by the models used during construction. Future, more advanced retrievers might find relevant products that are currently unjudged.

Additionally, the benchmark is currently focused on a zero-shot setting in Chinese. Future work could explore fine-tuning models specifically for intent-to-product reasoning and expanding the benchmark to a multilingual context.

Summary updated

Abstract

Real-world e-commerce queries often express user intent without naming a target product, and a single intent may be fulfilled by products from multiple categories. However, existing multimodal retrieval benchmarks typically assume a relatively well-defined target, leaving retrieval for intents with multi-category solutions less explored. To fill this gap, we introduce MIPR, a benchmark for Multimodal Intent-to-Product Retrieval comprising 3,522 multimodal queries and a shared corpus of 30,731 products. MIPR features reasoning-intensive retrieval, multi-category solutions, and fine-grained relevance labels. Experiments show that the best embedding-based retriever achieves 49.23% nDCG@10 on the full query set, and 55.86% solution-category recall@10 on queries with multiple solution categories. Reranking improves these scores to 60.11% and 67.00%, respectively, yet substantial gaps to the candidate-pool oracle remain. These results establish MIPR as a challenging and diagnostic benchmark for evaluating multimodal intent-to-product retrieval. The data and code are publicly available at https://huggingface.co/datasets/leequeue/MIPR.
BibTeX
@misc{li2026mipr,
  author = {Qiang Li and Xuan Lu and Jian Ruan and Manxi Lin and Jiacheng Chen and Wen Shen and Zhihua Wei and Xiaoyu Shen and Dan Ou and Xiangheng Kong and Junjun Zheng},
  title = {{MIPR: A Benchmark for Reasoning-Intensive Multimodal Intent-to-Product Retrieval with Multi-Category Solutions}},
  year = {2026},
  eprint = {2611.22509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22509},
}
Figure