Abstract
Function call capabilities have become crucial for Large Language Models (LLMs), enabling them to interact more effectively with external tools and APIs. Existing methods for improving the function call capabilities of LLMs rely on data obtained either through manual annotation or automated generation by models, and use this data to finetune the LLMs. However, these methods often lack targeted design and are constrained by fixed patterns and data distributions, which limits their effectiveness in enhancing the generalization and robustness of function call LLMs. To address this limitation, we propose a novel adversarial data augmentation method that employs reinforcement learning to systematically identify and target the weaknesses of function call LLMs. Our training framework introduces a query model trained with reinforcement learning (RL) to generate adversarial queries that are specifically designed to challenge function call (FC) models. This approach adopts a zero sum game formulation, where the query model and the FC model engage in iterative alternating training. Overall, our method advances the development of more robust FC models and provides a systematic way to identify and correct weaknesses in the ability of LLMs to interact with external tools.
Introduction
Modern LLMs act as "brains" for digital systems by calling external tools and APIs. However, training them to do this reliably is difficult. Current methods rely on manual annotation—which is slow and lacks scale—or standard LLM-generated data, which tends to be repetitive and follows predictable patterns.
The core issue is that these datasets rarely contain the "edge cases" or "failure modes" where models actually struggle. If a model only sees "easy" examples during training, it remains brittle when faced with complex or slightly malformed user queries in the real world. This research aims to move from static data generation to a dynamic, adversarial approach that actively explores a model's specific weaknesses.
Methodology
The proposed framework utilizes an iterative, alternating training process based on Zero-Sum Game Theory. The system consists of two primary agents:
- The Query Model (The Attacker): An RL-trained model that takes "seed" queries and rewrites them to be more challenging. Its goal is to maximize a reward based on making the FC model fail while still remaining semantically valid.
- The FC Model (The Defender): The model being trained to perform function calls. It is fine-tuned on the "bad cases" discovered by the Query Model.
To ensure quality and diversity, the framework incorporates several key components:
- Two-Stage Filtering: A larger Judge Model (Qwen3-32B) verifies that rewritten queries remain plausible and map to the correct tool.
- Embedding Loss: A regularization term rewards the Query Model for generating diverse queries, preventing "mode collapse" where the model repeats a single successful trick.
- Curriculum Learning: Training complexity increases gradually from simple single-function calls to complex parallel calls.
The models were trained using a seed dataset of ~1,700 manually annotated samples and augmented with 300 parallel-type samples from the xlam-function-calling-60k dataset in later rounds.
Results and Discussion
The adversarial approach was tested across various model sizes, including Qwen3-0.6B, 4B, 8B, and Qwen2.5-3B. Evaluation focused on the Berkeley Function-Calling Leaderboard (BFCL) metrics, including Non-Live AST accuracy and Relevance/Irrelevance detection.
Key findings include:
- Significant Performance Gains: Qwen3-0.6B improved its overall accuracy from 39.27% to 44.21%. Qwen2.5-3B-Instruct saw a jump from 43.82% to 49.87%.
- Iterative Improvement: Performance sometimes dipped after the first round as the model encountered a new, harder data distribution, but surged in the second round, validating the iterative training approach.
- Small Model Robustness: Smaller models (0.6B and 3B) benefited most significantly, making this approach ideal for enhancing on-device reliability where capacity is limited.
- Superiority over Standard Augmentation: In nearly all head-to-head tests, adversarial RL-augmented data outperformed data generated by standard LLM sampling, proving that targeted difficulty is more effective than random variety.
Conclusion
This work demonstrates that reinforcement learning can be effectively used as an exploratory tool to harden function-calling capabilities in LLMs. By treating the identification of model weaknesses as a search problem, the adversarial framework generates more diverse and challenging datasets than traditional methods. While larger models (8B+) sometimes exhibited "reward hacking" by exploiting formatting quirks, the overall results show a clear path toward more robust tool-use agents. Future research will expand this adversarial approach to multi-step reasoning and long-context tool interactions.
Summary updated