Abstract
As large language models (LLMs) advance, their inability to autonomously execute tasks by directly interacting with external tools remains a critical limitation. Traditional methods rely on inputting tool descriptions as context, which is constrained by context length and requires separate, often inefficient, retrieval mechanisms. We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM's parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation. Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents.
Introduction: The Bottleneck of Tool Interaction
Large Language Models (LLMs) are increasingly capable of complex reasoning, yet they remain largely isolated from the real world. To perform actionable tasks—such as booking travel or querying live weather data—they must interact with external APIs. Current methodologies for tool use generally follow two paths, both of which face significant scaling challenges:
- In-context Learning (ICL): Tool descriptions are prepended to the user prompt. This approach is limited by the model's context window; as the number of available tools grows into the thousands, it becomes impossible to fit all descriptions into a single prompt.
- Retrieval-Augmented Generation (RAG): A separate retriever model identifies a subset of relevant tools, which are then passed to the LLM. This introduces "error propagation"—if the retriever fails, the agent cannot recover—and adds significant latency to the interaction.
ToolGen proposes a fundamental shift: instead of reading about tools, the LLM should "speak" them. By treating tools as unique tokens within the model's vocabulary, ToolGen unifies retrieval and execution into a single generative process.
Methodology: Tool Virtualization and Training
The core of ToolGen is Tool Virtualization. Rather than describing an API with natural language, the researchers expand the LLM’s vocabulary (using Llama-3-8B as a base) by adding approximately 47,000 "virtual tokens." Each token corresponds to a unique API.
The Three-Stage Training Pipeline
To embed this tool knowledge into the model's weights, the authors developed a three-stage training strategy:
- Tool Memorization: The model is fine-tuned to predict a tool's unique token when presented with its documentation. This step ensures the model understands the semantic purpose of each virtual token.
- Retrieval Training: The model learns to generate the correct tool token in response to diverse, natural language user queries.
- End-to-End Agent Tuning: The model is trained on complete trajectories, including reasoning (Thought), tool invocation (Action Token), parameter generation, and processing observations to reach a final answer.
Atomic Indexing and Constrained Search
ToolGen utilizes Atomic Indexing, where each tool is represented by a single, distinct token. This avoids the "length bias" found in semantic indexing (where longer tool names might receive lower probabilities during beam search). To ensure reliability, the system employs Constrained Beam Search during inference. This restricts the model's output space to only valid tool tokens, effectively eliminating the possibility of hallucinating non-existent tools.
Experimental Results
The framework was evaluated using ToolBench, a massive dataset featuring over 47,000 unique APIs. To maintain general conversational capabilities, the researchers integrated OpenHermes-2.5 to create "ToolGen-Instruct."
Retrieval Performance
In multi-domain retrieval tasks involving all 47,000 tools, ToolGen achieved an NDCG@1 of 87.67. This significantly outperforms traditional retrievers like BM25 and BERT-based ToolRetriever (which scored 72.31). By internalizing tool knowledge, the model identifies relevant tools more accurately than external search mechanisms.
Task Completion
On the Solvable Pass Rate (SoPR) metric within the StableToolBench framework, ToolGen-Instruct reached 58.84%. This performance exceeds that of GPT-3.5 and specialized baselines like ToolLlama. The results demonstrate that unifying retrieval and calling reduces the friction and error rates inherent in multi-step agentic workflows.
Discussion and Future Directions
ToolGen represents a move toward "first-class" tool integration in LLMs. By representing tools as tokens, the model treats API calls with the same predictive fluency as natural language.
However, challenges remain. Unlike RAG systems where a database can be updated instantly, adding new tools to ToolGen currently requires fine-tuning. Future research will focus on continual learning techniques to update the tool vocabulary more efficiently and the integration of Reinforcement Learning (RL) to further optimize tool selection and complex reasoning trajectories.
By transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and truly autonomous AI systems.
Summary updated