TL;DR
IntentLens is a two-stage "ground-then-rank" framework that uses tool-augmented Multimodal Large Language Models (MLLMs) to resolve vague, underspecified queries (e.g., "Find a place with this vibe"). By orchestrating visual, memory, and item-attribute tools, it transforms ambiguous inputs into precise intent representations, significantly outperforming traditional retrieval and zero-shot recommendation baselines.
What Problem Does This Paper Address?
Imagine you’re wearing smart glasses and point your camera at a cozy, dimly lit café. You ask, "Find me something with this vibe, but more affordable and open late."
To a human, this is clear. To a traditional recommendation system, it’s a nightmare. Current systems typically fall into two camps:
- Passive Matchers: They fuse image and text features but struggle with "fuzzy" language and implicit constraints.
- Text-Heavy Recommenders: They expect fully specified textual queries and can't effectively "see" the visual nuances (like color palette or lighting) that define a "vibe."
The authors identify a gap in intent grounding: the ability to actively gather evidence from the image, the user's history, and item metadata to fill in the blanks of an underspecified query.
Methods
The authors propose IntentLens, which operates in a two-stage ground-then-rank pipeline:
Stage 1: Tool-Augmented Grounding
Instead of directly matching the query to items, an MLLM (like Qwen2.5-VL) acts as an agent. It follows a "Plan -> Observe -> Ground" loop:
- Visual Tools: Extract quantitative cues like color histograms, brightness, or specific crops from the query image.
- User Memory Tools: Retrieve relevant snippets from the user’s past interactions (e.g., "The user likes quiet places with natural light").
- Item Attribute Tools: Fetch fine-grained metadata for candidates, such as specific review insights or real-time operating hours.
The MLLM synthesizes this evidence into a grounded query—a detailed textual description of what the user actually wants.
Stage 2: Efficient Ranking
Because MLLM reasoning is computationally expensive, IntentLens decouples it from the final scoring. A lightweight dual-encoder ranker (based on BGE-M3) takes the grounded query and enriched item descriptions to produce a final Top-K list.
Training: Ranker-Aligned RFT
To teach the MLLM which tools actually help, the authors use Reinforcement Fine-Tuning (RFT). They employ a counterfactual ranking reward: the model gets a higher reward if the evidence it gathered actually improves the final rank of the "ground truth" item compared to a baseline with no tools. They optimize this using Group Relative Policy Optimization (GRPO), a memory-efficient RL algorithm.
Datasets
The authors introduced two new benchmarks specifically designed for underspecified multimodal queries:
- GOOGLE REVIEW-MMTOOL: 773 queries across 1,556 categories.
- YELP-MMTOOL: 1,727 queries across 497 categories.
These benchmarks categorize queries into four complexity tiers based on the tools required for resolution (e.g., Visual only vs. Visual + Memory + Item).
Key Insights and Ideas
- Reasoning vs. Representation: The "aha moment" here is that for underspecified queries, reasoning about the intent is more important than just having a better embedding. Tools allow the model to "quantify" the vibe (e.g., color temperature in Kelvin) rather than relying on holistic, often noisy, visual embeddings.
- Decoupling for Scalability: By separating tool-based grounding from ranking, the system can handle large candidate pools efficiently. The MLLM only "thinks" about the query once, while the ranker handles the heavy lifting of comparing thousands of items.
- Counterfactual Rewards: Training the agent to value useful evidence prevents "tool sprawl"—where a model calls every tool available regardless of need—leading to more precise and faster planning.
Key Results
IntentLens (using a 7B parameter backbone) achieved state-of-the-art results:
- Superior Accuracy: On Yelp-MMTool, it achieved a Hit@10 of 70.4% and an MRR of 46.7, significantly outperforming GPT-4o (Hit@10: 66.2, MRR: 41.4) and specialized multimodal recommenders like UniMP.
- Human Preference: In a blinded study, human annotators preferred IntentLens’s recommendations over BGE Retrieval on 89% of queries.
- Generalization: The model remained robust when tested on unseen item categories and across different datasets, proving that the tool-using logic is transferable.
Limitations and Future Work
- Latency: The full pipeline takes about 4 seconds per query. While tolerable for some apps, it’s a bottleneck for real-time "always-on" wearable experiences.
- Single-Turn Focus: The current work focuses on one-shot recommendations. In the real world, users might want to refine their intent through multi-turn dialogue.
- Privacy: Accessing a user's full interaction history and camera feed raises significant privacy concerns, which the authors acknowledge will require robust on-device safeguards in production.
For more details, see the full paper: "IntentLens: Grounding Underspecified Multimodal Queries for Recommendation via Tool-Augmented Reasoning" (NeurIPS 2026).
Summary updated