RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intent

Xiao Chen, Yicheng Zhao, Yingying Wu, Zhendong Chu, Changyi Ma, Qingsong Wen, Xuan SongarXiv 2026 · arXiv

TL;DR

RecToolBench is a new benchmark designed to evaluate how well AI agents can navigate the "fuzzy" reality of human recommendations—where users are vague and tasks require complex tool orchestration. The study reveals a significant "syntactic-semantic gap": while modern LLMs are great at following API schemas, they frequently fail to infer correct parameters from underspecified user intent, especially as tasks grow in complexity.

What Problem Does This Paper Address?

We are currently witnessing a shift from passive recommender systems (which just filter lists) to agentic recommenders—AI agents that use external tools (like search engines, database filters, or sentiment analyzers) to solve a user's request.

However, existing benchmarks for these agents have three major flaws:

  1. Explicit Intent Bias: They assume users provide perfectly structured instructions, whereas real humans are often vague ("I want something like that Sony camera I saw once").
  2. Simplified Environments: They often test isolated function calls rather than the complex, multi-step "orchestration" required for real-world tasks.
  3. The Syntactic-Semantic Gap: Most benchmarks only check if a tool call is valid (syntactic success). They don't check if the parameters used actually align with the user's latent needs (semantic success).

Methods

The authors introduce RecToolBench, built on the Model Context Protocol (MCP), a standardized framework that allows agents to interact with various tool servers seamlessly.

To create realistic challenges, they developed the AmbiFlow Synthesis pipeline:

  • Synthesize: Generate a clear, executable task based on real-world user-item interaction data.
  • Fuzzify: Inject five types of ambiguity: underspecification, soft contradiction, vague quantifiers, preference uncertainty, and implicit constraints.
  • Judge: A quality-filtering stage to ensure the tasks remain solvable despite the fuzziness.

The benchmark categorizes tasks into four levels of increasing difficulty, as shown in the pipeline overview:

Figure 1: The overall data synthesis and agent evaluation pipeline, featuring a progressive four-level difficulty scheme. Tailored to address inherently fuzzy user instructions, the task complexity escalates from easy to hard: L1 requires a single tool call; L2 involves parallel tool invocations; L3 demands serial dependency chains; and L4 entails hybrid dependencies and tool orchestration.

Evaluation is two-pronged: Rule-based metrics (checking if the tool ran and if the final item was found) and Rubric-based LLM-as-a-Judge (using a strong model like DeepSeek-V3 to score the agent's planning coherence and grounding).

Datasets

RecToolBench utilizes three diverse recommendation domains:

  • Amazon Electronics: Product search and accessory recommendation.
  • Yelp 2019: Restaurant discovery and review analysis.
  • Foursquare: Location-based services and spatial reasoning.

The benchmark consists of 1,288 executable tasks involving 13 MCP servers and 32 unique tools. The distribution of these tools and tasks is visualized below:

Figure 2: Overview of MCP server and task distribution in the RecToolBench.

Key Insights and Ideas

  • Parameter Grounding is the Bottleneck: The biggest failure point isn't calling the tool; it's figuring out what to put in the tool call. Agents struggle to translate "cheap" into a specific min_price or max_price parameter that leads to the correct result.
  • The "Smart" Strategy Gap: Proprietary models (like GPT-4 or Claude) often use "auxiliary" tools—like fetching item details—to gather evidence before making a final recommendation. Small Language Models (SLMs) tend to be more rigid, sticking to the bare minimum toolset and failing more often.
  • Redundancy vs. Efficiency: Many open-source models achieve high "Parallel Rates" (calling many tools at once) but have low "Efficiency Scores" because their calls are redundant or repetitive rather than strategically decomposed.
  • Syntactic Success $\neq$ Recommendation Success: In complex hybrid tasks (L4), models often achieve high schema compliance (90%+) but dismal Hit Ratios (~20-28% for SLMs). They are "playing the part" of an agent without actually solving the user's problem.

Key Results

  • Atomic Ability (L1): Proprietary models achieve near 100% tool selection accuracy. The best SLM (Qwen-2.5-7B) reaches only 58% accuracy and 45% parameter accuracy.
  • Sequential Orchestration (L3): Even when models select the right tools, they fail at "Dependency Awareness"—carrying information from Step 1 to Step 2.
  • Hybrid Complexity (L4): This is the "frontier" of the benchmark. Even the best proprietary models see their performance drop significantly, with Hit Ratios falling to between 39% and 54%.
  • Model Size Isn't Everything: The researchers found that instruction-tuning and tool-use alignment are often more important than raw parameter count; for example, Llama-3.1-8B performed comparably to larger models in specific tool-selection tasks.

Limitations and Future Work

The authors acknowledge that RecToolBench operates in a controlled MCP environment, which doesn't fully account for the latency and noise of production-grade industrial systems. Furthermore, the study only covers three domains; expanding to more diverse tool ecosystems and user behaviors is a clear next step. Finally, due to cost, the researchers couldn't explore the full spectrum of agent architectures (like multi-agent debate or complex reflection loops), leaving room for future exploration into which agent designs best handle fuzzy intent.

Summary updated

Abstract

Recent advances in agentic recommender systems are shifting recommender systems from passive filtering engines to instruction-following agents that use external tools to resolve user intent. However, existing benchmarks often assume explicit user intent, simplified tool environments, or isolated function calls, leaving realistic tool orchestration for recommendation underexplored. To bridge this gap, we propose RecToolBench, a Model Context Protocol (MCP)-based benchmark for evaluating tool-using recommender agents under fuzzy user instructions. RecToolBench contains more than 1,200 executable tasks across three recommendation domains, 13 MCP servers, and 32 tools, spanning single-tool calls, parallel tool calls, sequential tool chains, and hybrid tool orchestration. We construct RecToolBench with a scalable synthesize--fuzzify--judge pipeline that generates executable fuzzy recommendation tasks, and evaluates agent trajectories using rule-based execution checks and rubric-based LLM evaluation. Experiments on representative LLMs show that syntactically valid tool calls do not guarantee successful recommendations. Models struggle with semantic parameter grounding, multi-step evidence integration, and grounded final recommendations, especially as orchestration complexity increases. Our results identify tool orchestration under fuzzy user intent as a major bottleneck for agentic recommender systems. Our data and code are available at https://github.com/ShawnChenn/RecToolBench.
BibTeX
@misc{chen2026rectoolbench,
  author = {Xiao Chen and Yicheng Zhao and Yingying Wu and Zhendong Chu and Changyi Ma and Qingsong Wen and Xuan Song},
  title = {{RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intent}},
  year = {2026},
  eprint = {2609.30717},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.30717},
}
Figure