TL;DR
LLMs often struggle with tool use because their training data is either too simple or contains hallucinations from larger "teacher" models. This paper introduces AETool, an adversarial framework where a query-generation model and a tool-use model co-evolve, iteratively pushing each other to handle increasingly complex, high-quality, and diverse tasks without relying on massive external datasets.
What Problem Does This Paper Address?
While Large Language Models (LLMs) are powerful, they are essentially "frozen" after pre-training. To act as agents, they must use external tools (APIs) to access real-time data or perform complex computations. Currently, the most common way to teach LLMs tool use is through supervised fine-tuning (SFT) on synthetic data generated by frontier models like GPT-4.
However, this approach has two major flaws:
- Data Quality: Even the best models hallucinate, leading to synthetic training sets with subtle errors.
- Difficulty Mismatch: If the training data is too easy, the model doesn't learn; if it’s too hard, the model suffers from "negative transfer," where its original knowledge structure is disrupted.
The authors identify a "research gap" in applying self-play or adversarial evolution—techniques that have revolutionized LLM performance in math and coding—specifically to the domain of tool use.
Methods
The authors propose AETool (Antagonistic Evolution for Tool use). The framework sets up a "cat-and-mouse" game between two models: a Query-Generation Model ($M_{gen}$) and a Tool-Use Model ($M_{tool}$).
The process follows three main stages:
- Warmup: Since small base models aren't naturally good at generating complex queries, the authors first fine-tune $M_{gen}$ to rewrite low-quality, simple queries into fluent, structurally sound ones.
- Antagonistic Evolution (The Core Loop):
- Updating the Generator: $M_{gen}$ creates multiple candidate queries. $M_{tool}$ attempts to solve them and provides a confidence score. $M_{gen}$ is then trained to turn "easy" queries (high confidence) into "hard" queries (low confidence).
- Updating the Solver: $M_{tool}$ is then trained on these newly minted, more challenging queries to improve its accuracy.
- Data Mutation: To prevent the model from simply memorizing a static mapping of tools to solutions, the authors introduce a reparameterization strategy. They perturb the tool arguments (e.g., changing a city name in a weather API call) and force the generator to create a new, matching natural language query.
This adversarial loop ensures that the training data is always at the "frontier" of what the model can currently handle.
Datasets
The models were trained using 10,000 samples from the xlam-function-calling-60k and ToolACE datasets.
For evaluation, the authors used three major benchmarks:
- BFCL-live (Berkeley Function Calling Leaderboard): The primary benchmark for real-world tool use.
- API-Bank: Evaluates the model across different levels of API interaction complexity.
- ACEBench: A benchmark focusing on diverse tool-use scenarios, including single-turn and multi-turn interactions.
Key Insights and Ideas
- Confidence as a Difficulty Proxy: The paper uses the model’s own confidence scores to identify the "learning boundary." By training the generator to minimize the solver's confidence, they ensure the solver is always being challenged by tasks it almost can't do.
- Escaping the SFT Plateau: Standard SFT on static data often leads to overfitting, where performance peaks early and then declines. AETool shows a near-linear improvement trend because the data distribution evolves alongside the model’s capabilities.
- Generality Across Scales: The method isn't just for large models. The authors demonstrated that even 1.5B and 3B parameter models see significant boosts in tool-use capability when trained this way.
Key Results
AETool consistently outperformed both standard SFT and specialized state-of-the-art tool-use models:
- BFCL-live: AETool achieved an average score of 0.8068, significantly higher than the SFT baseline (0.7772) and other models like ToolACE-8B (0.7602) and Watt-tool-8B (0.7650).
- API-Bank & ACEBench: The model showed superior performance across all levels, particularly in "Multiple" and "Parallel" tool-use categories, where logic and parameter extraction are most difficult.
- Data Complexity: In a "Data Complexity Study," the authors showed that as the iterations progressed, the accuracy of all models (even Llama 3.1) dropped on the generated data, proving that the generator was successfully creating objectively harder tasks.
Limitations and Future Work
- Resource Constraints: Due to limited hardware, the authors used LoRA (Low-Rank Adaptation) for fine-tuning rather than full-parameter updates.
- Warmup Dependency: The success of the evolution depends heavily on the initial "warmup" phase; if the initial generator is too weak, the adversarial loop may struggle to start.
- Future Directions: While not explicitly detailed, the framework opens the door for integrating real-world execution feedback (e.g., whether the API call actually worked) into the adversarial loop to further reduce hallucinations.
Summary updated