Large Language Models (LLMs) have largely mastered the syntax of single-step tool invocation. Given a clear intent and a well-defined API documentation string, modern models can reliably generate the correct function call and extract the necessary parameters.
However, real-world tasks rarely yield to a single API call. Solving complex user requests requires multi-step tool orchestration—the ability to compose, sequence, and execute a series of distinct tools over multiple turns. When transitioning from isolated tool selection to sequential orchestration, even advanced models experience a sharp decline in performance. Understanding why this breakdown occurs, and how researchers are fixing it, requires looking closely at how data and intent flow through a multi-step plan.
The Core Bottleneck: Parameter Inheritance and Data Flow
The primary reason sequential tool use fails is not a lack of syntax mastery, but a failure to manage inter-tool dependencies. In a multi-step workflow, tools do not operate in a vacuum; they form a chain where the output of Tool A serves as the necessary input for Tool B.
[User Query] ──> [ Tool A ] ──(Intermediate Output)──> [ Tool B ] ──> [Final Answer]
Liu et al. demonstrate that while models are adept at identifying which tools to use, they frequently fail to maintain the data flow across sequential steps. This phenomenon is often framed as a failure of parameter inheritance (Wang et al.). For example, if a model must look up a user's order ID via a database tool and then pass that specific ID to a shipping calculator, it must parse the unstructured or semi-structured output of the first tool, isolate the correct variable, and inject it precisely into the next tool's schema.
When the output of a tool is complex or unexpected, the data flow breaks. Min et al. note that managing this high-level data flow becomes exponentially more difficult as the chain lengthens, leading to a compounding rate of execution errors.
The Disconnect Between Planning and Execution
Multi-step orchestration exposes a fundamental gap in LLM capabilities: the disparity between low-level function calling and high-level workflow planning.
As Ahn et al. observe, an agent can be perfectly capable of formatting a valid API call while remaining completely blind to the global coherence of the task. When forced to transition from simple information retrieval to complex tool composition, models often lose track of their long-horizon goals. Evaluation frameworks like ToolQA (Zhuang et al.) show that success rates can plummet to single digits when a task requires organizing multiple specialized tools (such as database loaders, calculators, and interpreters) in a precise logical sequence.
This planning bottleneck is further exacerbated by the "fuzzy" or ambiguous nature of real-world instructions. As Wang highlights, models struggle to decompose vague objectives into concrete sequential steps, often failing to recognize when operations can be executed in parallel versus when they must wait for a prior dependency to resolve.
From Linear Chains to Directed Acyclic Graphs
Standard autoregressive language models naturally bias toward linear, step-by-step generation (such as standard Chain-of-Thought prompting). However, complex real-world workflows are rarely strictly linear. They often require branching logic, parallel execution tracks, or conditional loops.
To bridge this structural mismatch, recent research shifts away from text-based linear planning toward modeling tool dependencies explicitly as Directed Acyclic Graphs (DAGs).
┌──> [ Tool B (Extract Text) ] ──┐
[ User Query ] ───┤ ├──> [ Tool D (Synthesize) ]
└──> [ Tool C (Fetch Image) ] ───┘
By forcing the model to generate or reason over a plan structured as a DAG, the agent can explicitly map out which tools depend on which outputs before execution begins. Lu et al. utilize this graph-based paradigm to maintain global coherence across multi-turn interactions, using graph-based rewards to train models to respect complex dependency structures. Similarly, Yin et al. demonstrate that translating function dependencies into explicit graphs allows for the synthesis of highly consistent multi-turn training data, preventing the state-tracking failures that typically plague long-horizon agentic workflows.
Inverting the Paradigm: Backward Reasoning
When humans plan a complex journey, we often work backward from the destination to determine the immediate next step. Traditional agent frameworks, however, force LLMs to reason exclusively forward, predicting the first tool call based only on the initial prompt.
Zhang et al. challenge this forward-only convention with the Reverse Chain framework. Instead of asking "What tool should I use first?", the model reasons backward from the final goal. It identifies the final API required to produce the ultimate answer, determines what arguments that API requires, and then recursively searches for prior APIs capable of generating those arguments.
This inversion fundamentally simplifies the parameter filling problem. Because the model works backward from required schemas, the expected outputs of preceding tools are already strictly constrained by the inputs of subsequent tools, eliminating much of the guesswork and hallucination associated with forward-looking planning.
Dynamic Adaptation and Execution Feedback
Even with a perfect initial plan, static orchestration fails when real-world APIs return errors, unexpected schemas, or empty results. Robust multi-step orchestration therefore requires a tight coupling between internal reasoning and external execution feedback.
Rather than relying purely on pre-computed plans, modern architectures embed verification loops directly into the execution cycle:
- Self-Verification: Before committing to a tool sequence or parameter assignment, models can use contrastive questioning to evaluate alternative paths. Mekala et al. show that introducing a self-verification step allows models to generalize much more effectively to unfamiliar tools by catching selection errors before they propagate down the dependency chain.
- Introspection and Perplexity: Models can monitor their own internal uncertainty during generation. The framework introduced by Gao et al. uses token perplexity as a signal for "introspection," allowing the system to detect when it is confused by a complex tool sequence and dynamically adjust its trajectory.
- Reinforcement from Outcomes: Rather than just cloning human behavior, models can be trained via reinforcement learning based on the actual success or failure of the execution chain. As demonstrated by Qiao et al., integrating execution feedback teaches the model not just how to call a tool, but when to rely on external tools versus its own internal knowledge, leading to cleaner, less redundant orchestration chains.
Key Foundational Work
ToolQA: A Dataset for LLM Question Answering with External Tools (2023) — contributed 4 concept elements to this direction.
An Evaluation Mechanism of LLM-based Agents on Manipulating APIs (2024) — contributed 2 concept elements to this direction.
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (2023) — contributed 2 concept elements to this direction.