Introduction
Large Language Model (LLM) agents have demonstrated remarkable proficiency in simple, isolated tasks. However, their performance often degrades when applied to complex, multi-step real-world scenarios. This research identifies two primary bottlenecks in current agentic workflows: methodological brittleness and coarse evaluation metrics.
Most existing frameworks rely on "static" planning, where a task is decomposed into a fixed sequence of steps at the outset. If an early step fails or yields unexpected data, the agent typically proceeds to the next step regardless, resulting in Cascading Task Failure (CTF). Furthermore, many systems utilize a fixed library of "hard-coded" subagents that lack the flexibility to adapt to diverse or unpredictable requests. To address these issues, we propose TDAG (Dynamic Task Decomposition and Agent Generation), a framework that treats planning as a living process and generates specialized subagents on the fly.
The TDAG Framework
The TDAG framework introduces a fluid operational model centered on three core innovations:
- Dynamic Decomposition: Unlike static checklists, the TDAG "Main Agent" maintains a dynamic plan. After each subtask is executed, the framework re-evaluates and updates the remaining subtasks based on the actual results or failures encountered. This allows the system to pivot and implement a "Plan B" when necessary.
- On-the-Fly Agent Generation: Rather than relying on pre-defined agents, TDAG generates specific subagents tailored to the immediate subtask. This process includes generating "cleaned-up" tool documentation, which reduces context noise and enhances the subagent's focus on the specific problem at hand.
- Incremental Skill Library: Drawing inspiration from autonomous agents like Voyager, TDAG maintains a library of successful solutions to previous subtasks. Using SentenceBERT, the system retrieves relevant past solutions to inform new subtasks, allowing the agent to learn from experience without manual intervention.
ItineraryBench: A New Standard for Evaluation
To rigorously test the framework, the authors introduced ItineraryBench, a benchmark focused on complex travel planning. It comprises 364 samples across 15 cities and 83 attractions, categorized into three levels of complexity: Inter-city planning (Type 1), Intra-city planning (Type 2), and combined personal assistant scenarios (Type 3).
A critical contribution of this work is the move away from binary "pass/fail" scoring. Binary metrics often mask an agent's true capabilities; for instance, an agent that successfully books three flights but fails to find a specific hotel is labeled a total failure. TDAG advocates for a fine-grained scoring system that evaluates agents based on three dimensions: Executability, Constraint Satisfaction, and Efficiency. This approach reveals that agents are often "mostly right," providing a more nuanced roadmap for future research.
Experimental Results
TDAG was evaluated against established baselines, including ReAct and ADAPT, across ItineraryBench and other environments like WebShop and TextCraft.
On ItineraryBench, TDAG achieved an average fine-grained score of 49.08, significantly outperforming ReAct (43.02) and ADAPT (44.74). The framework's ability to mitigate Cascading Task Failure was particularly notable: while static methods like Plan & Execute (P&E) suffered from CTF rates as high as 34.78%, TDAG reduced this to just 4.35%.
In the Minecraft-inspired TextCraft environment, TDAG reached a success rate of 73.5%, a substantial improvement over the previous state-of-the-art (ADAPT at 52.0%). Ablation studies confirmed that the "Dynamic Decomposition" component is the framework's primary driver of performance, as its removal led to a significant drop in success rates.
Limitations and Future Work
Despite its strengths, TDAG faces several challenges. First, skill verification in non-coding environments (like travel planning) relies on LLM self-summarization, which can introduce persistent errors if a skill is summarized incorrectly. Second, the constant re-planning and agent generation increase computational cost and latency compared to linear chains. Finally, while highly effective with GPT-4, the framework's performance on smaller, open-source models remains a subject for future investigation.
In conclusion, TDAG demonstrates that dynamic planning and specialized agent generation are essential for moving LLM agents beyond simple tasks and toward reliable, real-world utility.
Summary updated