IntentLens: Grounding Underspecified Multimodal Queries for Recommendation via Tool-Augmented Reasoning

Xiao Chen, Dong Fang, Haitao Li, Qing LiarXiv 2026 · arXiv

TL;DR

IntentLens is a two-stage "ground-then-rank" framework that uses tool-augmented Multimodal Large Language Models (MLLMs) to resolve vague, underspecified queries (e.g., "Find a place with this vibe"). By orchestrating visual, memory, and item-attribute tools, it transforms ambiguous inputs into precise intent representations, significantly outperforming traditional retrieval and zero-shot recommendation baselines.

What Problem Does This Paper Address?

Imagine you’re wearing smart glasses and point your camera at a cozy, dimly lit café. You ask, "Find me something with this vibe, but more affordable and open late."

To a human, this is clear. To a traditional recommendation system, it’s a nightmare. Current systems typically fall into two camps:

  1. Passive Matchers: They fuse image and text features but struggle with "fuzzy" language and implicit constraints.
  2. Text-Heavy Recommenders: They expect fully specified textual queries and can't effectively "see" the visual nuances (like color palette or lighting) that define a "vibe."

The authors identify a gap in intent grounding: the ability to actively gather evidence from the image, the user's history, and item metadata to fill in the blanks of an underspecified query.

Figure 1: IntentLens leverages three types of tools to disambiguate underspecified multimodal queries, producing a clarified user intent and fine-grained candidate item attributes through MLLM-based tool-augmented reasoning, thereby enabling more accurate recommendations.

Methods

The authors propose IntentLens, which operates in a two-stage ground-then-rank pipeline:

Stage 1: Tool-Augmented Grounding

Instead of directly matching the query to items, an MLLM (like Qwen2.5-VL) acts as an agent. It follows a "Plan -> Observe -> Ground" loop:

  • Visual Tools: Extract quantitative cues like color histograms, brightness, or specific crops from the query image.
  • User Memory Tools: Retrieve relevant snippets from the user’s past interactions (e.g., "The user likes quiet places with natural light").
  • Item Attribute Tools: Fetch fine-grained metadata for candidates, such as specific review insights or real-time operating hours.

The MLLM synthesizes this evidence into a grounded query—a detailed textual description of what the user actually wants.

Stage 2: Efficient Ranking

Because MLLM reasoning is computationally expensive, IntentLens decouples it from the final scoring. A lightweight dual-encoder ranker (based on BGE-M3) takes the grounded query and enriched item descriptions to produce a final Top-K list.

Training: Ranker-Aligned RFT

To teach the MLLM which tools actually help, the authors use Reinforcement Fine-Tuning (RFT). They employ a counterfactual ranking reward: the model gets a higher reward if the evidence it gathered actually improves the final rank of the "ground truth" item compared to a baseline with no tools. They optimize this using Group Relative Policy Optimization (GRPO), a memory-efficient RL algorithm.

Figure 2: The inference pipeline of INTENTLENS. The MLLM first performs tool-augmented reasoning to produce a clarified query representation and select relevant item-side tools. The selected tools are then applied to the candidate pool in a decoupled pass, yielding fine-grained item representations. Finally, a lightweight dual-encoder ranker scores candidates against the grounded query to enable efficient recommendations.

Datasets

The authors introduced two new benchmarks specifically designed for underspecified multimodal queries:

  • GOOGLE REVIEW-MMTOOL: 773 queries across 1,556 categories.
  • YELP-MMTOOL: 1,727 queries across 497 categories.

These benchmarks categorize queries into four complexity tiers based on the tools required for resolution (e.g., Visual only vs. Visual + Memory + Item).

Distribution of synthesized queries across difficulty tiers (T1–T4).

Key Insights and Ideas

  • Reasoning vs. Representation: The "aha moment" here is that for underspecified queries, reasoning about the intent is more important than just having a better embedding. Tools allow the model to "quantify" the vibe (e.g., color temperature in Kelvin) rather than relying on holistic, often noisy, visual embeddings.
  • Decoupling for Scalability: By separating tool-based grounding from ranking, the system can handle large candidate pools efficiently. The MLLM only "thinks" about the query once, while the ranker handles the heavy lifting of comparing thousands of items.
  • Counterfactual Rewards: Training the agent to value useful evidence prevents "tool sprawl"—where a model calls every tool available regardless of need—leading to more precise and faster planning.

Key Results

IntentLens (using a 7B parameter backbone) achieved state-of-the-art results:

  • Superior Accuracy: On Yelp-MMTool, it achieved a Hit@10 of 70.4% and an MRR of 46.7, significantly outperforming GPT-4o (Hit@10: 66.2, MRR: 41.4) and specialized multimodal recommenders like UniMP.
  • Human Preference: In a blinded study, human annotators preferred IntentLens’s recommendations over BGE Retrieval on 89% of queries.
  • Generalization: The model remained robust when tested on unseen item categories and across different datasets, proving that the tool-using logic is transferable.

Limitations and Future Work

  • Latency: The full pipeline takes about 4 seconds per query. While tolerable for some apps, it’s a bottleneck for real-time "always-on" wearable experiences.
  • Single-Turn Focus: The current work focuses on one-shot recommendations. In the real world, users might want to refine their intent through multi-turn dialogue.
  • Privacy: Accessing a user's full interaction history and camera feed raises significant privacy concerns, which the authors acknowledge will require robust on-device safeguards in production.

For more details, see the full paper: "IntentLens: Grounding Underspecified Multimodal Queries for Recommendation via Tool-Augmented Reasoning" (NeurIPS 2026).

Summary updated

Abstract

As intelligent personal assistants are increasingly deployed in real-world scenarios, recommendation paradigms are shifting from passive preference matching to inferring user intent from underspecified multimodal inputs. In practical interactions, users typically issue a reference image paired with a vague natural language query, keeping key visual attributes, personalized preferences, and item-side constraints implicit. Existing recommendation methods either leverage historical user interactions augmented with multimodal item features or assume fully articulated textual queries; both paradigms struggle to resolve underspecified multimodal queries for accurate item matching. To address this limitation, we propose IntentLens, a tool-augmented recommendation framework that grounds ambiguous multimodal queries into query-relevant evidence prior to user-item matching. IntentLens follows a ground-then-rank pipeline: a shared multimodal LLM leverages visual tools, user memory modules, and item attribute retrieval tools to (i) infer latent user intent from image-text inputs and (ii) augment each candidate item with fine-grained, query-conditioned evidence. Subsequently, a lightweight ranker scores candidates based on the grounded representations. To facilitate research in this novel setting, we construct two benchmarks with underspecified multimodal queries, namely \textsc{GoogleReview-MMTool} and \textsc{Yelp-MMTool}. Extensive experiments on these benchmarks demonstrate that IntentLens outperforms competitive MLLM and retrieval baselines, and exhibits strong generalization to unseen categories and datasets.
BibTeX
@misc{chen2026intentlens,
  author = {Xiao Chen and Dong Fang and Haitao Li and Qing Li},
  title = {{IntentLens: Grounding Underspecified Multimodal Queries for Recommendation via Tool-Augmented Reasoning}},
  year = {2026},
  eprint = {2611.11346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11346},
}
Figure