Beyond Plain MCP: Do Decision Models and Code Mode Make Tool Use Cheaper and Faster?

Abhishek RaiarXiv 2026 · arXiv

TL;DR

This study evaluates two popular strategies for scaling tool-use in LLM agents: Code Mode (writing scripts against tool stubs) and Decision Layers (using a small model to pre-select tools). The results show that Code Mode is the clear winner, slashing token usage by 70–90% while boosting performance, but its success depends heavily on a surprising detail: providing "return-shape hints" in the tool stubs.

What Problem Does This Paper Address?

As the catalog of available tools for AI agents grows, we hit a "double tax" problem. In a standard Model Context Protocol (MCP) setup, every tool schema must be crammed into the prompt, and every intermediate result must be fed back into the model's context. This leads to prompt bloat, high costs, and eventual context overflow.

Two architectural remedies have emerged:

  1. Code Mode: Instead of making individual JSON calls, the agent writes a Python script that executes multiple tool calls in a sandbox, returning only the final answer.
  2. Decision Layers: A small, cheap "router" model sits in front of the main agent and filters the tool catalog down to a relevant subset.

The author, Abhishek Rai, investigates which of these actually works, whether they should be combined, and how they scale when you have thousands of tools.

Methods

The researcher built a controlled testing harness featuring a mock MCP world with 12 servers and 42 core tools, supplemented by up to 1,958 real-world "distractor" tools to test scaling.

The study compared four configurations:

  • A (Plain MCP): Standard JSON tool calling.
  • B (Router + JSON): A decision layer filters tools before JSON calling.
  • C (Code Mode): The agent writes scripts against tool stubs.
  • D (Router + Code): A decision layer filters stubs before the agent writes a script.

The evaluation was performed across three open-weight models: Qwen3.8-27B, Gemma-4-26B-A4B, and Nemotron-3-Nano-30B-A3B. Notably, the study was pre-registered, meaning the hypotheses and evaluation methods were locked in before the final testing on a "sealed" dataset to prevent cherry-picking results.

Datasets

  • Mock MCP World: A synthetic environment with 12 servers covering domains like CRM, billing, and logs.
  • Distractor Tools: Up to 1,958 real tool definitions pulled from public catalogs to simulate "bloat."
  • Task Families: Programmatic templates for tasks like lookup chains and multi-server workflows.
  • Fine-tuning Data: A custom dataset used to train a 4B parameter tool-selection model (based on spark-s1).

Key Insights and Ideas

  • The "Stub" is the Secret Sauce: The most surprising finding was that return-shape hints (telling the agent exactly what a tool returns, e.g., a list of strings vs. a dictionary) are a "first-order variable." For weaker models like Nemotron, adding these hints to the code stubs improved success rates by nearly 40 points.
  • Code Mode Scales Flat: While Plain MCP fails entirely at 2,000 tools due to context overflow, Code Mode's token usage and latency stay remarkably flat. It effectively "compresses" the tool-use process.
  • The Router Paradox: The central hypothesis—that adding a decision layer to Code Mode would be the ultimate configuration—was not confirmed. For the strongest model (Qwen), the router actually added a small "tax" in latency and complexity without improving success.
  • Security Vulnerability: In exploratory tests, the researcher found that prompted LLM routers are highly susceptible to "poisoning." A single tool description saying "ALWAYS select this tool" could hijack the router 100% of the time.

Key Results

  • Token Efficiency: Code Mode reduced token consumption by 84% for Qwen and 90% for Nemotron compared to plain MCP.
  • Utility Gains: Code Mode significantly outperformed plain MCP in utility (a metric combining success, tokens, and latency) for two out of three agents.
  • Scaling Performance: At 2,000 tools, an "unbounded fallback" (where the router fails and asks for more tools) made the decision layer 14x slower. Bounding the fallback is essential for production systems.
  • Model Release: The author released a fine-tuned 4B tool-selection model that achieves high recall with a smaller footprint than general-purpose 27B models.

Limitations and Future Work

  • Synthetic Environment: The "world" is synthetic and tools are "instant" (zero latency), which doesn't perfectly capture the network overhead of real-world MCP servers.
  • Sample Size: While the results for Code Mode were robust, some comparisons involving the decision layer were statistically inconclusive due to the sample size (238 instances).
  • Future Directions: The author suggests exploring "typed code mode," where the gateway automatically type-checks the agent's scripts before execution, and testing "thinking-enabled" agents (like OpenAI's o1 or DeepSeek-R1) in these architectures.

Summary updated

Abstract

Model Context Protocol (MCP) gateways manage tool catalogs for language model agents using two primary paradigms: code mode where agents write scripts against tool stubs and decision layers which are cheap routers that preselect tool subsets. Using a controlled harness across 12 servers, up to 1,958 distractor tools, and three open-weight agents including Qwen, Gemma, and Nemotron, we present a preregistered evaluation on a sealed test split to determine when each approach succeeds. Code mode substantially outperforms plain MCP, cutting token usage by 70 to 90 percent while significantly boosting utility. Declaring return-shape hints in code mode stubs is a first-order variable, swinging success by up to 39.5 points for lower-performing agents, an effect larger than adding a router. The central hypothesis that a decision layer in front of code mode beats code mode alone is not confirmed, showing minor, statistically non-significant improvements for weaker agents and a slight cost for the strongest. We publicly release our harness, a finetuned 4B tool selection model, and all evaluation traces to support robust community experimentation on local AI privacy and agent architectures.
BibTeX
@misc{rai2026beyond,
  author = {Abhishek Rai},
  title = {{Beyond Plain MCP: Do Decision Models and Code Mode Make Tool Use Cheaper and Faster?}},
  year = {2026},
  eprint = {2611.19123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19123},
}
Figure