The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

TL;DR

The Tool Decathlon (TOOLATHLON) is a rigorous new benchmark designed to test language agents on diverse, multi-step, real-world tasks across 32 applications and 604 tools. The results serve as a wake-up call: even the most advanced model, Claude-4.5-Sonnet, succeeds only 38.6% of the time, highlighting a massive gap between conversational AI and reliable execution in complex software environments.

The Problem: Beyond the Chat Interface

While Large Language Models (LLMs) have become excellent at following instructions in a vacuum, turning them into "agents" that can actually perform work—like managing a WooCommerce store, auditing a Kubernetes cluster, or grading assignments in Canvas—remains difficult.

Existing benchmarks often suffer from three main flaws:

1. Narrow Scope: They focus on a single domain, such as web browsing or coding.
2. Lack of Realism: They use "mock" APIs or start from empty states (e.g., an empty inbox), which fails to reflect the messy, data-rich environments of real software.
3. Short Horizons: Tasks are often solved in 2–3 steps, whereas real-world workflows require dozens of interactions.

To build agents we can actually trust, we need a "Decathlon"—a benchmark that forces agents to navigate diverse tools, infer intent from ambiguous prompts, and maintain coherence over long trajectories.

Methods: Building the Decathlon

The researchers introduced TOOLATHLON, a framework built on the Model Context Protocol (MCP), an open standard for connecting AI to applications.

Expand

Source, p. 5

Key methodological innovations include:

• Containerized Environments: To ensure realism without the headache of managing hundreds of live cloud accounts, open-source versions of professional software are deployed inside isolated Docker containers.
• Realistic State Initialization: Unlike benchmarks that start with a "blank slate," TOOLATHLON tasks begin with pre-populated environments—such as a Canvas course with dozens of students.
• Fuzzy Prompting: Prompts are designed to be concise and ambiguous, forcing the agent to explore the environment to understand the exact requirements.
• Execution-Based Evaluation: Success is measured by whether the state of the software has actually changed correctly, rather than just evaluating the agent's text output.

The Benchmark: 32 Apps, 604 Tools

The primary contribution is the TOOLATHLON benchmark, which consists of:

• 32 Software Applications: Ranging from everyday tools (Google Calendar, Notion) to professional platforms (BigQuery, Kubernetes, Snowflake).
• 604 Tools: Sourced and refined from high-quality MCP servers.
• 108 Manually Crafted Tasks: These are "cross-app" by design, requiring the agent to coordinate between multiple platforms.

Expand

Source, p. 2

Key Insights and Agent Failures

Evaluating state-of-the-art models revealed several fascinating behavioral quirks:

• The "Laziness" Problem: In long-horizon tasks (like analyzing data over ten years), models often process the first year and then tell the user to apply the same steps for the rest.
• Observation vs. Internal Reasoning: Top-tier models actually use fewer tokens for internal reasoning than some "thinking" models, suggesting successful agents rely more on efficient environment observation than on circular internal monologues.
• Context Traps: Models frequently get "trapped" by overlong tool outputs. If a tool returns a massive HTML dump or database log, the agent often loses the thread of the original task.
• The Reliability Gap: There is a significant difference between a model being capable of a task and being reliable. Most models lack the consistency required for production-level automation.

Key Results

The benchmark proved exceptionally difficult for current state-of-the-art (SOTA) models:

• Claude-4.5-Sonnet took the top spot but only achieved a 38.6% success rate, averaging about 20 tool-calling turns per task.
• GPT-5 followed with 30.6%.
• DeepSeek-V3.2-Exp was the best-performing open-weights model, reaching 20.1%.
• Gemini-2.5-Flash struggled significantly, with only a 3.7% success rate, often giving up prematurely on complex tasks.

Limitations and Future Work

The authors acknowledge several areas for growth, including the ongoing evolution of MCP servers and the need for better error recovery. While models are getting better at identifying tool-calling errors, they still struggle to "debug" their own plans when an execution fails mid-way through a 20-turn trajectory. Future versions of the benchmark may also incorporate GUI-based interactions alongside API-based tool use.

By open-sourcing the environment and the evaluation scripts, TOOLATHLON aims to become the "North Star" for developing agents capable of handling the messy reality of professional workflows.

@inproceedings{li2026tool,
  author = {Junlong Li and Wenshuo Zhao and Jian Zhao and Weihao Zeng and Haoze Wu and Xiaochen Wang and Rui Ge and Yuxuan Cao and Yuzhen Huang and Wei Liu and Junteng Liu and Zhaochen Su and Yiyang Guo and Fan Zhou and Lueyang Zhang and Juan Michelini and Xingyao Wang and Xiang Yue and Shuyan Zhou and Graham Neubig and Junxian He},
  title = {{The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=z53s5p0qhf},
}