Abstract
Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM’s ‘imagination’ to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.
Introduction: The Tool-Use Reliability Gap
While the research community has successfully expanded the breadth of tool-augmented LLMs—enabling models to interface with thousands of APIs—a significant gap remains in the depth of their mastery. Current state-of-the-art models, including GPT-4, often fail to use tools accurately, with correctness rates frequently hovering between 30% and 60%. These failures typically stem from a misunderstanding of fine-grained API requirements, such as syntax constraints or specific argument formats.
In-Context Learning (ICL) often proves too shallow for complex APIs, and high-quality fine-tuning data is expensive to curate. Models frequently hallucinate arguments or fail to follow the strict logic required for successful execution. To bridge this gap, we introduce a framework that allows models to learn from their own mistakes in a simulated environment.
Methodology: Simulated Trial and Error (STE)
The Simulated Trial and Error (STE) framework is a biologically inspired approach that mimics how intelligent agents learn through practice and reflection. The framework operates in two distinct phases: Exploration and Exploitation.
1. Exploration (The Imaginarium)
In this phase, the LLM uses its "imagination" to generate plausible user queries based on an API's documentation. It then attempts to solve these queries through a Trial and Error process using a ReAct-style loop (Thought $\rightarrow$ Action $\rightarrow$ Observation). Crucially, the model learns from real execution feedback, such as error codes, to iteratively refine its calls.
To ensure effective learning, STE employs two memory mechanisms:
- Short-term Memory: Tracks recent trial trajectories within an episode to help the model correct immediate mistakes.
- Long-term Memory: Maintains a record of all previously explored queries across episodes. This acts as a diversity engine, preventing the model from repeating simple queries and forcing it to explore edge cases and diverse API functionalities.
2. Exploitation
Once a diverse set of successful "imagined" trajectories is collected, they are filtered and paraphrased to create a high-quality dataset. This data is then used to either provide superior demonstration examples for ICL or to fine-tune smaller open-source models for specialized tool mastery.
Experimental Results
Experiments conducted on ToolBench demonstrate the efficacy of STE across various model sizes and settings:
- Performance Leap: Mistral-Instruct-7B, which initially achieved only 30.1% correctness, jumped to 76.8% after STE fine-tuning, notably surpassing the GPT-4 baseline of 60.8%.
- ICL Enhancement: STE-generated examples improved GPT-4’s own performance from 60.8% to 76.3% when used as few-shot demonstrations.
- Continual Learning: By implementing an "experience replay" strategy—rehearsing a small portion of previous tool data while learning new ones—the models successfully mitigated catastrophic forgetting, maintaining performance on both old tools and general language benchmarks like MMLU.
Conclusion and Future Work
STE demonstrates that the performance of tool-augmented LLMs is limited more by the quality of training data than by model size. By allowing models to "practice" in a sandbox and learn from execution feedback, we can achieve high reliability even in modest 7B parameter models. Future research will focus on extending this framework to multi-tool composition and dynamic environments where API ground truths may change over time.
Summary updated