CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions

Zishan Guo, Yufei Huang, Deyi XiongACL 2024 · DOI

Abstract

CToolEval is a large-scale benchmark designed to evaluate Large Language Models (LLMs) as agents within the Chinese digital ecosystem. By testing 11 models across 398 real-world APIs from 27 popular Chinese applications, we identify a significant performance gap between proprietary and open-source models. Our framework introduces a two-stage evaluation process and a rollback mechanism to rigorously assess tool invocation and task completion capabilities.

Introduction

As Large Language Models (LLMs) evolve from conversational interfaces into autonomous agents, their ability to interact with the real world via APIs becomes paramount. Tasks such as booking travel, managing office workflows, or navigating e-commerce platforms require models to not only understand natural language but also to execute precise API calls and process dynamic feedback.

However, existing benchmarks often fall short by relying on synthetic environments or focusing exclusively on English-language contexts. This leaves a gap in understanding how LLMs perform within specific cultural and digital ecosystems, such as China's. CToolEval addresses this by providing a benchmark grounded in the real-world Chinese digital landscape, testing models on their ability to navigate the complexities of actual API interactions.

Methodology and Framework

The researchers developed a fine-grained evaluation framework that moves beyond simple success metrics to analyze the nuances of agent behavior. This framework is designed to be reproducible and rigorous, addressing the limitations of subjective scoring.

The CToolEval pipeline for dataset construction and evaluation, highlighting the interaction between users, agents, and real-world tools.

Two-Stage Evaluation

The framework evaluates models on two distinct levels:

  1. Tool Invocation Capability: Assessing whether the model correctly identifies the necessary tool and provides the appropriate parameters required by the API.
  2. Task Completion Capability: Evaluating if the model can synthesize the tool's output to fulfill the user's original request accurately.

Interaction Types and Rollback Mechanism

Queries are categorized into four types: Fixed-answer (static facts), Open-ended (descriptive), Real-time (dynamic data like weather), and Operational (state-changing actions). To maintain a clean testing environment for Operational tasks—such as adding a friend on a social platform—the system employs a Rollback Mechanism that automatically reverses the action after the test is completed.

The CToolEval Dataset

CToolEval is specifically curated for the Chinese societal context, offering a scale and scope previously unavailable for agent evaluation. It focuses on the tools that define the daily digital experience in China.

A sunburst chart showing the 14 domains and 27 popular Chinese applications included in the CToolEval benchmark.

The dataset includes 6,816 total instances, split nearly evenly between single-tool and complex multi-tool scenarios. It covers 14 domains, ranging from shopping and travel to office automation, utilizing APIs from major platforms like Taobao, Meituan, Baidu Maps, and Feishu.

Key Insights and Results

The evaluation of 11 models, including GPT-3.5-turbo and various open-source Chinese LLMs, yielded several critical findings:

  • The Performance Gap: GPT-3.5-turbo remains the gold standard, achieving a 61.94% tool matching rate. InternLM-chat-20B emerged as the strongest open-source performer (21.02%), though it still lags significantly in execution accuracy.
  • Tool Hallucination: A major hurdle for open-source models is "tool hallucination," where models fabricate server responses rather than correctly calling the API. This "deceptive agent" behavior makes them unreliable for real-world deployment.
  • Knowledge Conflict and Reasoning: Models often struggle when tool outputs contradict their internal training data. Furthermore, temporal reasoning remains a challenge; for instance, GPT-3.5-turbo frequently fails to calculate the correct date when a user mentions "tomorrow" in a query.
  • Fine-tuning Trade-offs: While fine-tuning on CToolEval data improves tool invocation scores, it can paradoxically increase "response hallucinations," where the model becomes better at calling the tool but worse at accurately reporting the results to the user.

Conclusion

CToolEval highlights the significant challenges remaining in the development of reliable AI agents. While proprietary models lead the way, open-source models require substantial improvements in groundedness and logical reasoning to effectively navigate real-world API interactions. Future work will focus on enhancing model reliability and maintaining the benchmark against the volatile nature of public APIs.

Summary updated

BibTeX
@inproceedings{guo2024ctooleval,
  author = {Zishan Guo and Yufei Huang and Deyi Xiong},
  title = {{CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.928},
}
Figure