Abstract
The rapid rise of Large Language Models (LLMs)-based intelligent agents underscores the need for robust, scalable evaluation frameworks. Existing methods rely on static benchmarks and labor-intensive data collection, limiting practical assessment. We introduce MCPEval, an open-source Model Context Protocol (MCP)-based framework that automates end-to-end task generation and deep evaluation of LLM agents across diverse domains. MCPEval standardizes metrics, seamlessly integrates with native agent tools, and eliminates manual effort in building evaluation pipelines. Empirical results across five real-world domains show its effectiveness in revealing nuanced, domain-specific performance. We publicly release MCPEval to promote reproducible and standardized LLM agent evaluation.
1. Introduction
As Large Language Model (LLM) agents evolve from simple conversational interfaces into autonomous systems capable of utilizing external tools, traditional evaluation methodologies have become insufficient. Existing benchmarks are frequently static, risking data contamination where models may have encountered evaluation tasks during training. Furthermore, constructing new benchmarks for complex, multi-step agentic workflows requires significant manual effort to define ground truth trajectories and verify solvability.
Standardization remains a critical hurdle; the lack of a unified protocol for agent-environment interaction makes fair comparisons across diverse tools and domains nearly impossible. MCPEval addresses this "evaluation gap" by leveraging the Model Context Protocol (MCP) to automate the creation, verification, and scoring of agentic tasks at scale.
2. The MCPEval Framework
The core innovation of MCPEval is its deep integration with the Model Context Protocol (MCP), a standard governing interactions between LLMs and external systems. The framework implements a three-stage automated pipeline:
- Task Generation: A specialized "Task-LLM" analyzes tool specifications from an MCP server to generate realistic user goals and instructions tailored to specific capabilities.
- Task Verification: To ensure solvability, a high-performing "Frontier LLM" acts as a gold-standard agent to execute the generated tasks. Successful executions establish the Ground Truth Trajectory, while failures trigger iterative task refinement.
- Deep Evaluation: Models under test are evaluated through two primary metrics: Tool Call Matching, which compares parameter-level tool usage against the ground truth, and LLM Judging, a rubric-based assessment of high-level planning and output quality.
3. Experimental Methodology
We evaluated 10 distinct models, including seven proprietary OpenAI models and three open-source alternatives (Qwen and Mistral). Rather than relying on static datasets, MCPEval generated over 10,000 individual task evaluations across five diverse real-world domains: Healthcare (clinical trials and PubMed), Airbnb (property searches), Sports (statistics), National Parks (visitor services), and YFinance (market analysis).
4. Results and Analysis
4.1 The Execution-Completion Gap
A primary finding of our study is the "execution-completion gap." Most models demonstrate higher proficiency in trajectory execution (correctly identifying and calling tools) than in completion (synthesizing tool outputs into a final, user-ready answer). This suggests that while models are becoming effective tool operators, they still struggle with the cognitive load of final result synthesis. Notably, OpenAI’s o3 model serves as an outlier, performing better at final completion than procedural consistency, suggesting a unique optimization for outcome-oriented reasoning.
4.2 Domain Sensitivity and Model Performance
Performance varied significantly by domain. Healthcare, characterized by highly structured data, yielded the highest success rates, whereas National Parks proved challenging due to complex geographical data integration. In terms of model ranking, o3 achieved the highest overall performance (0.926), followed by gpt-4o-mini (0.852). A persistent performance gap of approximately 0.148 points remains between top proprietary models and current open-source alternatives.
4.3 Bottlenecks in Tool Usage
Detailed analysis identified precise parameter specification as the primary bottleneck. While models often select the correct tool name, they frequently fail to provide the exact parameters required for successful execution. We observed a strong correlation (r=0.852) between tool name prediction and parameter matching, suggesting these skills are highly interdependent.
5. Conclusion
MCPEval provides a scalable, automated solution for the deep evaluation of AI agents. By moving beyond static benchmarks and utilizing standardized protocols like MCP, the framework reveals critical performance nuances such as the execution-completion gap. Future work will focus on incorporating non-synthetic, real-world task data and developing more cost-effective verification and judging strategies to further enhance scalability.
Summary updated