Abstract
Agentic tasks, which require multi-step problem solving with autonomy, tool use, and adaptive reasoning, are becoming increasingly central to the advancement of NLP and AI. However, existing instruction data lacks tool interaction, and current agentic benchmarks rely on costly human annotation, limiting their scalability. We introduce TaskCraft, an automated workflow for generating difficulty-scalable, multi-tool, and verifiable agentic tasks with execution trajectories. TaskCraft expands atomic tasks using depth-based and width-based extensions to create structurally and hierarchically complex challenges. Empirical results show that these tasks improve prompt optimization in the generation workflow and enhance supervised fine-tuning of agentic foundation models. We present a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.
Introduction
The AI field is rapidly shifting from passive chatbots to "agentic" systems—models that can autonomously use tools to solve complex, multi-step problems. However, training these agents is bottlenecked by data. Existing instruction-following datasets are mostly static text and lack tool interaction. Meanwhile, high-quality agentic benchmarks like GAIA or "Humanity's Last Exam" (HLE) rely on expensive, slow human annotation. TaskCraft addresses this scalability crisis by using Large Language Models (LLMs) to automatically build a massive library of complex, verifiable tasks that genuinely require tool use.
Methodology
The TaskCraft workflow operates in three main stages designed to ensure both complexity and verifiability.
1. Atomic Task Generation
The system starts with an unlabeled corpus (webpages, PDFs, images). It extracts a specific "input index" (like a document title), identifies a key fact within that context, and prompts an LLM to generate a question that can only be answered by accessing that specific tool context. This ensures that the task is grounded in real-world data and requires external tool invocation.
2. Task Extension and Scaling Difficulty
To move beyond simple one-step questions, TaskCraft uses two primary strategies:
- Depth-based Extension: This creates "n-hop" dependency chains. For example, if Task A is "Find the director of Movie X," the system might find a document stating "Movie X was released on Date Y" and transform the task into: "Who directed the film released on Date Y?"
- Width-based Extension: This merges multiple independent subtasks into a single complex query, requiring the agent to parallelize its tool use and synthesize information from multiple sources.
3. Agentic Execution and Verification
To ensure quality, the framework uses a "rejection sampling" strategy. It verifies that an LLM cannot answer the question using internal knowledge alone, ensuring tool necessity. Furthermore, the system generates execution trajectories to provide a ground-truth path for training.
The TaskCraft Dataset
The authors introduced the TaskCraft Dataset, a large-scale synthetic collection of approximately 36,000 tasks. The composition is diverse, with 75% web-based tasks, 15% image-based, and 10% PDF-based. These tasks cover critical domains including Finance, Academic Research, Government, and Culture. To optimize the generation process, the authors employed "bootstrap few-shot learning," which reduced generation time by nearly 20% while increasing the success rate to 68.1% for atomic tasks.
Key Results
- Performance Boost: Fine-tuning a base model (Qwen2.5-3B) on TaskCraft's synthetic trajectories led to massive gains. On the HotpotQA benchmark, performance jumped from a baseline of 0.032 to 0.232—a 7x improvement—after Supervised Fine-Tuning (SFT).
- Difficulty Alignment: The synthetic tasks successfully mirrored the difficulty levels of human-annotated benchmarks. Image-based tasks proved the most challenging, matching the failure rates seen in the most difficult levels of the GAIA benchmark.
- Tool Necessity: TaskCraft explicitly filters for tasks where a tool-less LLM fails but a tool-equipped agent succeeds, ensuring the dataset specifically targets agentic capabilities.
Conclusion
TaskCraft demonstrates that high-quality, complex agentic data can be generated at scale without the need for exhaustive human annotation. By recursively expanding tasks and implementing rigorous verification, the framework provides a viable path for training the next generation of autonomous AI agents. Future work will focus on expanding the toolset to include specialized environments like databases and coding sandboxes.
Summary updated