Planning and Editing What You Retrieve for Enhanced Tool Learning

Tenghao Huang, Dongwon Jung, Muhao ChenarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and generation, yet their utility in real-world applications often depends on their ability to interact with external tools such as APIs, calculators, and search engines. Current systems typically employ a "Retrieve-and-Read" strategy, where a system performs a one-time search for relevant tools based on a user's initial query. However, this approach is often insufficient for complex tasks due to two primary challenges: the Complexity Gap and the Semantic Gap.

The Complexity Gap arises when a single user query requires a sequence of interdependent tools that simple similarity matching cannot identify. The Semantic Gap occurs because tool documentation is frequently written by developers for developers—using abstract or technical language that does not align with how a typical user describes a task. To address these issues, we introduce PLUTO (Planning, Learning, and Understanding for TOols), a framework designed to bridge these gaps through iterative planning and dynamic documentation editing.

The PLUTO Framework

PLUTO shifts the paradigm from static retrieval to a dynamic process involving two core components: Plan-and-Retrieve (P&R) and Edit-and-Ground (E&G).

1. Plan-and-Retrieve (P&R)

Rather than attempting to find all necessary tools in a single pass, PLUTO utilizes an LLM-based Planner to decompose complex queries into actionable sub-tasks. For each sub-task, the system follows a structured loop:

  • Sub-query Generation: The Planner generates focused sub-queries that capture specific steps of the problem.
  • Neural Retrieval: A retriever (such as DPR or Contriever) identifies candidate tools for each sub-query.
  • Refinement and Prediction: Candidates are re-ranked using an "LM-likelihood" score, which measures the semantic fit between the tool description and the sub-query. Finally, an LLM-based Predictor selects the most relevant tools for the task.

Crucially, this process is execution-free. Unlike methods that require running a tool to determine the next step, PLUTO plans based on the LLM’s internal knowledge, significantly reducing computational latency and cost.

2. Edit-and-Ground (E&G)

PLUTO treats tool documentation as a learnable interface rather than a static ground truth. The Edit-and-Ground phase identifies tools that the retriever frequently fails to find despite their relevance.

Using an LLM, PLUTO analyzes successful user queries and instances of retrieval failure to "re-write" and enrich tool descriptions. By incorporating real-world scenarios and grounding the documentation in user-centric language, the framework makes these tools more discoverable for the neural retriever in future sessions.

Experimental Evaluation

The researchers evaluated PLUTO using the ToolBench dataset, a rigorous benchmark containing over 3,000 real-world APIs across 49 categories. The evaluation focused on the "Intra-Category" setting, which involves multi-tool scenarios with an average solution path of four tools per query.

Key Results

PLUTO demonstrated superior performance across multiple metrics compared to existing state-of-the-art (SOTA) models:

  • Retrieval Recall: Using the Contriever model in a non-finetuned setting, PLUTO achieved a 46.57% Recall, a significant improvement over the baseline's 31.78%.
  • Ranking Quality: The framework reached an 82.93 NDCG score, outperforming the ToolRetriever baseline of 74.70.
  • Execution Success: In end-to-end problem-solving tests, PLUTO achieved a 72.3% pass rate, surpassing the previous SOTA of 69.3%.
  • Model Agnostic: The framework proved effective across various LLMs (including ChatGPT and Llama2) and different retrievers, highlighting its versatility as a plug-and-play solution.

Discussion and Limitations

PLUTO demonstrates that planning is not merely an execution strategy but a superior method for retrieval. By breaking queries into specific sub-tasks, the system generates more effective search terms. Furthermore, the success of the E&G phase suggests that the "Semantic Gap" can be mitigated by allowing AI to optimize the very documentation it uses to find tools.

Despite these gains, some limitations remain. The current study was conducted exclusively on English-language datasets; different linguistic structures may present unique challenges for the planning and editing phases. Additionally, the E&G phase involves multiple LLM calls, which introduces an upfront computational cost. Future work will focus on expanding the framework to multilingual settings and developing automated criteria to optimize the cost-performance balance of tool description editing.

Conclusion

PLUTO represents a significant step forward in tool-augmented LLMs. By combining iterative planning with dynamic documentation grounding, the framework ensures that LLMs can more accurately find and utilize the tools necessary to solve complex, multi-step real-world problems.

Summary updated

Abstract

Recent advancements in integrating external tools with Large Language Models (LLMs) have opened new frontiers, with applications in mathematical reasoning, code generators, and smart assistants. However, existing methods, relying on simple one-time retrieval strategies, fall short on effectively and accurately shortlisting relevant tools. This paper introduces a novel PLUTO (Planning, Learning, and Understanding for TOols) approach, encompassing `Plan-and-Retrieve (P&R)` and `Edit-and-Ground (E&G)` paradigms. The P&R paradigm consists of a neural retrieval module for shortlisting relevant tools and an LLM-based query planner that decomposes complex queries into actionable tasks, enhancing the effectiveness of tool utilization. The E&G paradigm utilizes LLMs to enrich tool descriptions based on user scenarios, bridging the gap between user queries and tool functionalities. Experiment results demonstrate that these paradigms significantly improve the recall and NDCG in tool retrieval tasks, significantly surpassing current state-of-the-art models.
BibTeX
@misc{huang2024planning,
  author = {Tenghao Huang and Dongwon Jung and Muhao Chen},
  title = {{Planning and Editing What You Retrieve for Enhanced Tool Learning}},
  year = {2024},
  eprint = {2404.00450},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.00450},
}
Figure