MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models

Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang, Caiming Xiong

Abstract: The rapid adoption of Large Language Models (LLMs) as intelligent agents has underscored the necessity for robust evaluation frameworks capable of assessing agent performance in realistic, interactive environments. Existing evaluation methodologies often suffer from limitations such as static task benchmarks, limited scope, and inadequate integration with practical applications. In response, we introduce MCPEval, an open-source, Model Context Protocol (MCP)-based evaluation framework specifically tailored for comprehensive and systematic assessment of LLM-powered agents. MCPEval standardizes evaluations across diverse domains through automated task generation and verification, supports multiple performance metrics, and integrates seamlessly with native agent capabilities. We empirically validate the effectiveness of MCPEval across five distinct real-world domains, highlighting significant variations in performance across various LLM architectures and prompting strategies. Our results illustrate the framework’s capacity to uncover nuanced performance patterns and identify domain-specific strengths and weaknesses, providing valuable insights beyond traditional binary success metrics. We publicly release MCPEval to foster reproducible research and promote standardized evaluation practices within the LLM agent community.

Introduction

The evolution of Large Language Models (LLMs) from conversational interfaces to autonomous agents capable of interacting with external tools has created a significant evaluation bottleneck. Unlike standard text generation, agentic tasks require reasoning, multi-step planning, and precise tool invocation within dynamic environments. Traditional benchmarks often rely on static datasets that fail to capture the complexity of real-world workflows or require prohibitive amounts of manual labor to maintain.

To address these challenges, researchers from Salesforce AI Research introduced MCPEval at EMNLP 2025. MCPEval is an open-source framework designed to automate the "deep" evaluation of AI agents. By leveraging the emerging Model Context Protocol (MCP), the framework provides a standardized and scalable methodology for assessing whether an agent is truly mastering its environment or merely approximating success.

The MCPEval Framework

The core innovation of MCPEval lies in its integration with the Model Context Protocol (MCP), a standard for governing interactions between LLMs and external systems. This allows for a unified interface regardless of the underlying model or toolset. The framework operates through a three-stage automated pipeline:

  1. Task Generation: A specialized "Task-LLM" analyzes the specifications of available tools via MCP to generate complex, multi-step instructions that test the limits of an agent's reasoning.
  2. Verification and Ground Truth: To ensure generated tasks are solvable, a high-performing "frontier agent" (e.g., GPT-4o) executes the task. Successful executions are recorded as "Ground Truth Trajectories," while failures trigger a feedback loop to refine the task instructions.
  3. Deep Evaluation: Once a model-under-test completes a task, it is evaluated through two distinct lenses: Tool Call Matching (a quantitative check of parameter accuracy and sequence) and LLM Judging (a qualitative assessment of planning, execution flow, and adaptability).
The MCPEval automated task generation and verification pipeline, illustrating the flow from MCP server tool definitions to verified ground truth trajectories.

Experimental Setup

The effectiveness of MCPEval was validated by testing 10 state-of-the-art models, including the OpenAI O3 and GPT-4 series, alongside open-source models like Qwen and Mistral. The evaluation spanned five diverse real-world domains:

  • Healthcare: Clinical trials, drug information, and medical terminology.
  • Airbnb: Property search and booking simulations.
  • Sports: League standings, schedules, and player statistics.
  • National Parks: Facility booking and visitor services.
  • YFinance: Real-time financial data and stock analysis.

The resulting dataset comprised over 5,000 trajectory and completion records, offering a comprehensive look at the current state of LLM tool-use capabilities.

Results and Analysis

The empirical results revealed several critical insights into agent performance. OpenAI models consistently led the rankings, with the O3 model achieving the highest overall LLM judge score of 0.926. However, a significant performance gap remains between proprietary and open-source models, with the former averaging a score of 0.824 compared to 0.676 for the latter.

Comparative performance analysis across domains, highlighting the ranking of domains by average score and the persistent gap between trajectory execution and final task completion.

The "O3" Anomaly and Maturity Gaps

A notable finding was the "negative gap" observed in the O3 model. While O3 occasionally deviated from the reference tool-calling trajectory—leading to lower tool-match scores—it frequently produced higher-quality final results. This suggests that advanced reasoning models can discover more efficient paths than the provided ground truth.

Furthermore, the data highlighted a consistent "maturity gap" across most models: agents are generally more proficient at executing tool calls (trajectory) than they are at synthesizing that information into a final, user-ready report (completion). Domain difficulty was also found to be highly dependent on API design; Healthcare proved the most accessible due to standardized terminology, while National Parks presented the greatest challenge due to varied API structures and vocabularies.

Conclusion

MCPEval represents a significant step toward standardized, automated, and deep evaluation for AI agents. By moving beyond binary success metrics and leveraging the Model Context Protocol, the framework provides researchers with the tools to identify nuanced strengths and weaknesses in agentic behavior. Future work will focus on mitigating the costs and potential biases of LLM-based judging and incorporating more "messy," human-generated task data to further bridge the gap between synthetic benchmarks and real-world application.

Summary updated

Abstract

The rapid adoption of Large Language Models (LLMs) as intelligent agents has underscored the necessity for robust evaluation frameworks capable of assessing agent performance in realistic, interactive environments. Existing evaluation methodologies often suffer from limitations such as static task benchmarks, limited scope, and inadequate integration with practical applications. In response, we introduce MCPEval, an open-source, Model Context Protocol (MCP)-based evaluation framework specifically tailored for comprehensive and systematic assessment of LLM-powered agents. MCPEval standardizes evaluations across diverse domains through automated task generation and verification, supports multiple performance metrics, and integrates seamlessly with native agent capabilities. We empirically validate the effectiveness of MCPEval across five distinct real-world domains, highlighting significant variations in performance across various LLM architectures and prompting strategies. Our results illustrate the framework’s capacity to uncover nuanced performance patterns and identify domain-specific strengths and weaknesses, providing valuable insights beyond traditional binary success metrics. We publicly release MCPEval to foster reproducible research and promote standardized evaluation practices within the LLM agent community.
BibTeX
@inproceedings{liu2025mcpeval,
  author = {Zhiwei Liu and Jielin Qiu and Shiyu Wang and Jianguo Zhang and Zuxin Liu and Roshan Ram and Haolin Chen and Weiran Yao and Shelby Heinecke and Silvio Savarese and Huan Wang and Caiming Xiong},
  title = {{MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=l3TAq9TqD1},
}
Figure