In the world of e-commerce, we’ve gotten very good at "target-centric" search. If you type "blue Nike running shoes," modern algorithms will find them in milliseconds. But real-world shopping is often messier. Sometimes we have a problem, not a product name. We might snap a photo of a rusty kitchen knife and ask, "How can I make this look new again?"
A new paper introduces MIPR, a benchmark designed to test whether AI can handle these "intent-centric" queries that require deep reasoning and diverse solutions.
TL;DR
MIPR is a new reasoning-intensive benchmark for multimodal e-commerce retrieval that requires models to infer user intent from image-text pairs. Unlike traditional benchmarks, it features multi-category solutions and fine-grained relevance labels, revealing that even state-of-the-art models struggle to find diverse products that satisfy complex user needs.
What Problem Does This Paper Address?
Existing multimodal retrieval benchmarks typically assume a well-defined target. However, real-world user needs are often open-ended. A single intent—like restoring a rusty knife—can be fulfilled by products from entirely different categories: a chemical rust remover, a specialized rust eraser, or even high-grit sandpaper.
As illustrated in Figure 1: An example of the intent-to-product retrieval task., current models face three major hurdles:
- Reasoning-Intensive Retrieval: The model must infer the intent (rust removal) rather than just matching keywords.
- Multi-Category Solutions: A single query might have valid solutions across multiple functional categories.
- Fine-Grained Constraints: Models must distinguish between "optimal" solutions (food-safe removers) and "suboptimal" or "constraint-violating" ones (industrial-grade chemicals).
Methods
The authors developed a sophisticated pipeline to build MIPR from real Chinese e-commerce search logs (Taobao):
- Query Curation: They filtered ~130k raw logs down to ~3.5k high-quality queries that require joint multimodal understanding and have objective solutions.
- Candidate Discovery: Using LLMs, they generated diverse "product terms" (e.g., "baking soda," "rust eraser") to ensure the candidate pool covered various solution types.
- Fine-Grained Annotation: They used a "council of LLMs" (including Qwen, GLM, and DeepSeek) with human adjudication to label products across five levels: L1 (Optimal), L2 (Suboptimal), N1 (Constraint-violating), N2 (Related non-solution), and N3 (Irrelevant).
- Pooling: To ensure the benchmark is robust, they merged the top results from ten different state-of-the-art retrievers and annotated any previously unjudged pairs.
Datasets
The MIPR (Multimodal Intent-to-Product Retrieval) benchmark includes:
- 3,522 multimodal queries (image + question).
- 30,731 products in a shared corpus.
- 447,809 fine-grained relevance judgments. The dataset is grounded in real-world search behavior and is publicly available on Hugging Face.
Key Insights and Ideas
- The Power of Both Modalities: Modality ablation studies show that using both image and text is essential. As shown in Figure 3: Modality ablation and the impact of explicit intent., multimodal queries significantly outperform text-only or image-only versions. The image provides the situational context, while the text defines the specific need.
- The Reasoning Gap: When the authors manually added the "inferred intent" to the query, performance jumped significantly. This suggests that the "bottleneck" for current models isn't just retrieval—it's the internal reasoning step required to bridge the gap from a problem to a solution.
- Hard Negatives are Truly Hard: Models frequently rank "related non-solutions" (N2) highly. For example, if you ask how to fix a knife, a model might suggest buying a new knife. While semantically related, it fails to fulfill the user's specific intent of restoration.
Key Results
- Performance Ceiling: Even the best embedding-based retriever (Qwen3-VL-Embedding) only achieved an nDCG@10 of 49.23%.
- Reranking is Better but Not Enough: Applying a specialized reranker improved the score to 60.11%, but a massive gap remains compared to the "oracle" upper bound (86.04%).
- Category Coverage: Models struggle to provide a diverse set of solutions. For queries with multiple valid categories, the best model only recalled 55.86% of those categories in its top 10 results. This "solution-category recall" drops sharply as the number of possible categories increases, as seen in Figure 5: Performance vs. number of solution categories..
Limitations and Future Work
The authors acknowledge that while their pooling-based approach is thorough, it is inherently limited to the products discovered by the models used during construction. Future, more advanced retrievers might find relevant products that are currently unjudged.
Additionally, the benchmark is currently focused on a zero-shot setting in Chinese. Future work could explore fine-tuning models specifically for intent-to-product reasoning and expanding the benchmark to a multilingual context.
Summary updated