Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models

Yuxiang Zhang, Xin Fan, Junjie Wang, Chongxian Chen, Fan Mo, Tetsuya Sakai, Hayato Yamana

Abstract

Recent advancements in large language models (LLMs) integrated with external tools and APIs have successfully addressed complex tasks by using in-context learning or fine-tuning. Despite this progress, the vast scale of tool retrieval remains challenging due to stringent input length constraints. In response, we propose a pre-retrieval strategy from an extensive repository, effectively framing the problem as the massive tool retrieval (MTR) task. We introduce the MTRB (massive tool retrieval benchmark) to evaluate real-world tool-augmented LLM scenarios with a large number of tools. This benchmark is designed for low-resource scenarios and includes a diverse collection of tools with descriptions refined for consistency and clarity. It consists of three subsets, each containing 90 test samples and 10 training samples. To handle the low-resource MTR task, we raise a new query-tool alignment (QTA) framework leverages LLMs to enhance query-tool alignment by rewriting user queries through ranking functions and the direct preference optimization (DPO) method. This approach consistently outperforms existing state-of-the-art models in top-5 and top-10 retrieval tasks across the MTRB benchmark, with improvements up to 93.28% based on the metric Sufficiency@k, which measures the adequacy of tool retrieval within the first k results. Furthermore, ablation studies validate the efficacy of our framework, highlighting its capacity to optimize performance even with limited annotated samples. Specifically, our framework achieves up to 78.53% performance improvement in Sufficiency@k with just a single annotated sample. Additionally, QTA exhibits strong cross-dataset generalizability, emphasizing its potential for real-world applications.

Introduction

As the ecosystem of available APIs and external tools for large language models (LLMs) expands into the thousands, the "context wall" becomes a significant barrier. It is no longer feasible to fit all tool descriptions into a single prompt for in-context learning. Current architectures typically solve this using a two-step process: a retrieval model narrows down the massive database to a manageable subset, which the LLM then processes.

However, this approach suffers from a persistent semantic gap. User queries are often vague and colloquial, whereas tool documentation is technical and structured. Furthermore, fine-tuning retrieval models for specific toolsets usually requires thousands of annotated examples—a luxury most developers do not have. To address these challenges, we introduce MTRB, a new benchmark for massive tool retrieval, and QTA, a framework that leverages reinforcement learning to align user queries with tool documentation.

The Query-Tool Alignment (QTA) Framework

The core innovation of the QTA framework is treating the LLM as a "translator" for the retrieval model. Rather than searching the database with the raw user query, the LLM (e.g., Mistral-7B) rewrites the query into a detailed, structured format that aligns with the technical language of tool documentation.

The QTA framework utilizes DPO to train an LLM rewriter that aligns user queries with a frozen retrieval model.

To train this rewriter efficiently, we use Direct Preference Optimization (DPO). The LLM generates multiple candidate rewrites, which are then ranked by a frozen retrieval model using a custom ranking function based on Discounted Cumulative Gain (DCG). If a rewrite successfully surfaces the "Golden Tools" (the correct tools for the task) in the top-k results, it is used as a positive training sample. This allows the LLM to learn a general strategy for "thinking like a retriever" with minimal data.

The Massive Tool Retrieval Benchmark (MTRB)

To evaluate performance in realistic, low-resource settings, we developed the Massive Tool Retrieval Benchmark (MTRB). It consolidates data from three major sources—RestBench, MetaTool, and ToolBench—covering up to 2,391 real-world APIs. Unlike previous benchmarks, MTRB is designed for extreme data efficiency, providing only 10 training samples per subset to test how well models can generalize from sparse information.

Results and Evaluation

Our experiments demonstrate that QTA significantly outperforms standard retrieval baselines. A key metric used is Sufficiency@k (S@k), a binary measure that returns 1 only if all required tools for a multi-step task are present in the top-k retrieved results.

Comparative performance of QTA against various retrieval baselines across the MTRB subsets.

On the RestBench subset, QTA achieved a 93.28% improvement in S@5 compared to the best fine-tuned baseline. Remarkably, the framework is highly effective even in "one-shot" scenarios; with just a single annotated training sample, QTA improved S@k by 78.53% over zero-shot baselines. This efficiency stems from the LLM's ability to leverage its internal knowledge to bridge the semantic gap once it understands the target documentation style.

Conclusion

The QTA framework represents a shift in tool retrieval strategy: instead of optimizing the retriever, we optimize the query. By using LLMs to bridge the semantic gap through DPO-based alignment, we achieve state-of-the-art performance in massive tool retrieval with minimal data requirements. Future work will explore co-training the rewriter and the retriever simultaneously to further enhance alignment in even larger-scale tool repositories.

Summary updated

BibTeX
@inproceedings{zhang2024data,
  author = {Yuxiang Zhang and Xin Fan and Junjie Wang and Chongxian Chen and Fan Mo and Tetsuya Sakai and Hayato Yamana},
  title = {{Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2024},
  doi = {10.1145/3673791.3698429},
}
Figure