On the Tool Manipulation Capability of Open-sourced Large Language Models

Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, Jian ZhangSubmission to ICLR 2024 · Rejected

TL;DR

While proprietary models like GPT-4 excel at turning natural language goals into executable API calls, open-source LLMs severely lag behind due to API hallucination, incorrect argument populating, and non-executable outputs. By combining programmatic data generation for instruction alignment, BM25-based in-context demonstration retrieval, and strict system prompts, the authors boost open-source LLMs by up to 90% in success rate—matching or beating GPT-4 on half of evaluated tasks with only about one developer-day of human supervision per tool.

What Problem Does This Paper Address?

Using LLMs to automate software via APIs (tool manipulation) usually relies on closed models like GPT-4. However, enterprises often avoid commercial APIs due to data privacy, security, and IP risks.

Switching to open-source alternatives out-of-the-box reveals an enormous performance gap: in one test, open-source models failed 100% of cases while GPT-4 succeeded 77% of the time. The authors analyze why open-source models fail and investigate whether they can achieve parity with leading closed LLMs under realistic, low-cost human supervision.

Methods

The authors identify three primary failure modes in open-source LLMs: wrong API selection, wrong argument population, and non-executable code generation (e.g., chatty responses). To resolve these, they adapt three techniques:

  1. Multi-Tool Alignment via Programmatic Data Curation: Instead of expensive manual labeling, developers write a few dozen templates of goals and corresponding API calls with placeholder variables (e.g., {position}, {height}). These are populated from random value pools to synthesize thousands of training samples.
  2. Demonstration Retrieval: A BM25 retriever selects top-$k$ relevant examples from a small human-curated pool ($O(n)$ examples for $n$ candidate APIs) and prepends them as in-context demonstrations to guide argument formatting.
  3. Generation Regulation with System Prompts: A structured prompt template provides inference-time instructions enforcing that the model emits only executable code, suppressing conversational filler.

Datasets

The authors introduce ToolBench, an open-source evaluation benchmark featuring 8 tools covering both single-step and multi-step action planning:

  • Newly Collected Tools (5): OpenWeather, The Cat API, Home Search, Trip Booking, and Google Sheets.
  • Derived from Prior Datasets (3): VirtualHome (household simulation), WebShop (e-commerce navigation), and Tabletop (robotic spatial planning).
  • Key Metric & Features: Evaluates real execution outcomes rather than exact string matches. The benchmark quantifies task difficulty via a novel API Complexity Score (measuring how far test calls are from demonstration combinations) and identifies tasks requiring advanced reasoning.

Key Insights and Ideas

  • Synthetic data does the heavy lifting: Ablations reveal that model alignment on templated synthetic data contributes the vast majority of performance gains, enabling models to internalize API signatures.
  • In-context retrieval enables unseen generalization: Providing retrieved few-shot examples allows models to generalize to API combinations that never appeared in the demonstration pool.
  • Standard instruction-tuning hurts tool usage: Open-source models instruction-tuned for standard NLP/chat tasks did not outperform their raw base pre-trained counterparts on tool manipulation.
  • Reasoning remains the bottleneck: Open-source models match or beat GPT-4 on API selection and direct argument extraction, but still struggle significantly when complex intermediate reasoning or environmental state extraction is required (e.g., WebShop button IDs or Google Sheets coordinate math).

Key Results

  • Out-of-the-Box Disparity: In zero-shot settings, open-source models (LLaMA-30B, StarCoder, CodeGen-16B) often scored near 0% on multi-API tasks (e.g., 0% vs. GPT-4's 91.5% on Trip Booking).
  • Parity with GPT-4: With the proposed recipe, open-source success rates jumped by up to 90%. Open models matched or outperformed GPT-4 on 4 out of 8 tasks (OpenWeather, The Cat API, VirtualHome, and WebShop).
  • Closed Model Gaps Narrowed: On Home Search, LLaMA-30B rose from 0% to 87.0% (GPT-4 scored 98.0%). On Trip Booking, it reached 85.8% (GPT-4 scored 99.2%).
  • Practical Supervision Cost: Curation required only $O(n)$ templates (typically $<100$) and a small demonstration bank, averaging roughly one developer-day per tool.

Limitations and Future Work

  • Advanced Reasoning Deficit: Open models still significantly lag behind GPT-4 on tasks requiring intermediate logical leaps and non-API code generation (e.g., Google Sheets: 21.2% vs. GPT-4's 68.6%; Tabletop: 16.2% vs. 83.8%).
  • Model Scale Dependency: Gains scaled heavily with model capacity and underlying code generation proficiency, indicating that smaller models (<15B) may struggle without stronger base foundations.
  • Future Work: Developing specialized pre-training or fine-tuning approaches targeted at stateful multi-step reasoning, complex grounded decision-making, and specialized retrievers tailored for action generators.

Summary updated

Abstract

Recent studies on software tool manipulation with large language models (LLMs) mostly rely on closed model APIs. The industrial adoption of these models is substantially constrained due to the security and robustness risks in exposing information to closed LLM API services. In this paper, we ask can we enhance open-source LLMs to be competitive to leading closed LLM APIs in tool manipulation, with practical amount of human supervision. By analyzing common tool manipulation failures, we first demonstrate that open-source LLMs may require training with usage examples, in-context demonstration and generation style regulation to resolve failures. These insights motivate us to revisit classical methods in LLM literature, and demonstrate that we can adapt them as model alignment with programmatic data generation, system prompts and in-context demonstration retrievers to enhance open-source LLMs for tool manipulation. To evaluate these techniques, we create ToolBench, a tool manipulation benchmark consisting of diverse software tools for real-world tasks. We demonstrate that our techniques can boost leading open-source LLMs by up to 94% success rate, showing capabilities competitive to OpenAI GPT-4 in 4 out of 8 ToolBench tasks. We show that such enhancement typically requires about one developer day to curate data for each tool, rendering a recipe with practical amount of human supervision.
BibTeX
@inproceedings{xu2023tool,
  author = {Qiantong Xu and Fenglu Hong and Bo Li and Changran Hu and Zhengyu Chen and Jian Zhang},
  title = {{On the Tool Manipulation Capability of Open-sourced Large Language Models}},
  booktitle = {FMDM @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=d5ogyvdl1X},
}
Figure