Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents

Zhenning Shi, Liang Pang, Yongqi LiarXiv 2026 · arXiv

TL;DR

This paper demonstrates that the "user interface" for an AI agent—how tools and feedback are represented—is not just a passive wrapper but a fundamental part of the decision-making process. By comparing the Model Context Protocol (MCP) and Command-Line Interfaces (CLI) under strictly matched capabilities, the authors show that representation alone can swing task performance by over 30 points and radically alter token costs.

What Problem Does This Paper Address?

In the world of language agents, we often debate whether a model is "good at tool use." However, tool use is a two-way street: it depends on the model's intelligence and the interface through which it interacts with the world.

The authors identify an "attribution problem" in current research. When we compare different agent systems, we often change the tools, the prompts, and the execution environment all at once. This makes it impossible to tell if a performance boost comes from a better model, a better protocol (like MCP), or just a more intuitive set of commands. This paper seeks to isolate the representational effect: if the underlying capabilities are identical, how much does the "look and feel" of the interface (e.g., JSON-based tool calls vs. string-based terminal commands) change agent behavior?

Methods

The researchers introduced two novel frameworks to study and improve agent-interface interaction:

  1. MICA (Matched Interface Comparison and Analysis): This is a controlled experimental setup. It uses a shared "semantic action contract"—a ground-truth definition of what an agent can do—and renders it into either MCP or CLI formats. This ensures that any difference in behavior is due to the representation, not the capability. MICA factorizes the interface into three components:
  2. Presentation ($\Phi$): How tools and arguments are described.
  3. Realization ($R$): The syntax used to invoke the tool (e.g., JSON vs. flags).
  4. Feedback ($\Psi$): How the environment's response is shown back to the model.
  5. DART (Decoupled Action Realization and Targeted Repair): Based on the finding that agents often know what they want to do but fail at the syntax of the interface, DART splits the process. First, the model selects a "semantic commitment" (the intent). Then, it tries to "realize" that intent in the specific interface format. If the syntax fails, DART triggers a "Targeted Repair" that fixes the formatting while keeping the original intent fixed, preventing the agent from getting confused and changing its plan entirely.
Overview of the MICA and DART frameworks.

Datasets

The study evaluated six different models (from the Qwen3, DeepSeek, and Gemini families) across three diverse interactive environments:

  • ALFWorld: Grounded household tasks (e.g., "put a cool mug in the coffee machine").
  • WebShop: Goal-directed online shopping.
  • ScienceWorld: Multi-step scientific reasoning and experimentation.

The analysis covered over 2,000 paired episodes to trace how small interface differences propagate into major trajectory divergences.

Key Insights and Ideas

  • Interfaces are Context: The interface isn't just a pipe; it's part of the prompt. Different representations provide different "cues" that shift the model's internal probability distribution for the next action.
  • The "First Divergence" is Decisive: In 94.3% of cases, MCP and CLI agents eventually took different paths. These divergences usually happened very early in the task. Once an agent makes a slightly different local decision due to an interface quirk, the history changes, and the two trajectories rarely meet again.
  • Failure Pathways Shift: CLI interfaces tend to fail more at the "decoding" stage (syntax errors), whereas MCP failures often happen later in the pipeline, such as when a tool is technically valid but unavailable in the current state.
Figure 2: Failure pathways under MCP and CLI. Bars show the composition of failure stages among unsuccessful attempts. The dominant failure pathway changes with both the interface and environment.

Key Results

  • Performance Reversals: There is no "one true interface." For Qwen3-14B, CLI crushed MCP on ALFWorld (+36.6 points), but MCP was superior on WebShop (+4.9 points).
  • Token Efficiency: Interface choice significantly impacts the "interaction tax." On ALFWorld, MCP used 54.9k more tokens per task than CLI, but on WebShop, it saved 35k tokens.
  • DART Effectiveness: By separating intent from syntax, DART improved benchmark scores across all six evaluated settings (up to +9.0 points) and drastically reduced token costs for MCP (up to 27.1k tokens per task) by preventing "retry loops" where the agent re-plans from scratch after a simple typo.

Limitations and Future Work

The authors acknowledge that their study is limited to the current popular paradigms of MCP and CLI. They suggest that future work should explore "agent-native" interfaces—representations designed specifically for how LLMs process information, rather than just adapting human-centric terminal or API structures. Additionally, while DART is effective, it adds an extra model call for the "commitment" phase, which the authors mitigated through token savings elsewhere, but remains a factor for real-time latency.

Summary updated

Abstract

Language agents increasingly act through external tools and interactive environments, making the interface between the model and its executable capabilities part of the decision process. Model Context Protocol (MCP) and command-line interfaces (CLI) can expose the same semantic operations through markedly different model-facing representations. We study whether these representational differences change agent behavior when executable capabilities are matched. We introduce Matched Interface Comparison and Analysis (MICA), which renders MCP and CLI from a shared semantic action contract and routes decoded actions through the same runtime. Across six models and three interactive environments, matched interfaces produce substantial differences in task performance and interaction cost. These effects extend beyond surface validity to semantic action selection, failure pathways, and recovery. Matched-state probes localize sensitivity to capability presentation, realization, and feedback. Paired natural trajectories reveal structured divergence in 94.3% of 2,004 episodes and show that early differences propagate through subsequent interaction. This analysis identifies interface realization as a direct intervention boundary. We therefore introduce Decoupled Action Realization and Targeted Repair (DART), which preserves validated semantic commitments while repairing interface-specific realization. DART improves benchmark score in all six evaluated settings by up to 9.0 points and reduces MCP token cost in all three environments by up to 27.1k tokens per task. Together, these results establish interface representation as part of the closed-loop decision environment shaping agent behavior.
BibTeX
@misc{shi2026matched,
  author = {Zhenning Shi and Liang Pang and Yongqi Li},
  title = {{Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents}},
  year = {2026},
  eprint = {2610.34298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34298},
}
Figure