TaskCraft: Automated Generation of Agentic Tasks

Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Liu, Jian Yang, Ge Zhang, Jiaheng Liu, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, Wangchunshu Zhou

Abstract

Agentic tasks, which require multi-step problem solving with autonomy, tool use, and adaptive reasoning, are becoming increasingly central to the advancement of NLP and AI. However, existing instruction data lacks tool interaction, and current agentic benchmarks rely on costly human annotation, limiting their scalability. We introduce TaskCraft, an automated workflow for generating difficulty-scalable, multi-tool, and verifiable agentic tasks with execution trajectories. TaskCraft expands atomic tasks using depth-based and width-based extensions to create structurally and hierarchically complex challenges. Empirical results show that these tasks improve prompt optimization in the generation workflow and enhance supervised fine-tuning of agentic foundation models. We present a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.

Introduction

The AI field is rapidly shifting from passive chatbots to "agentic" systems—models that can autonomously use tools to solve complex, multi-step problems. However, training these agents is bottlenecked by data. Existing instruction-following datasets are mostly static text and lack tool interaction. Meanwhile, high-quality agentic benchmarks like GAIA or "Humanity's Last Exam" (HLE) rely on expensive, slow human annotation. TaskCraft addresses this scalability crisis by using Large Language Models (LLMs) to automatically build a massive library of complex, verifiable tasks that genuinely require tool use.

Methodology

The TaskCraft workflow operates in three main stages designed to ensure both complexity and verifiability.

1. Atomic Task Generation

The system starts with an unlabeled corpus (webpages, PDFs, images). It extracts a specific "input index" (like a document title), identifies a key fact within that context, and prompts an LLM to generate a question that can only be answered by accessing that specific tool context. This ensures that the task is grounded in real-world data and requires external tool invocation.

The TaskCraft framework generates atomic tasks by extracting context from documents and identifying implicit relationships to formulate queries.

2. Task Extension and Scaling Difficulty

To move beyond simple one-step questions, TaskCraft uses two primary strategies:

  • Depth-based Extension: This creates "n-hop" dependency chains. For example, if Task A is "Find the director of Movie X," the system might find a document stating "Movie X was released on Date Y" and transform the task into: "Who directed the film released on Date Y?"
  • Width-based Extension: This merges multiple independent subtasks into a single complex query, requiring the agent to parallelize its tool use and synthesize information from multiple sources.

3. Agentic Execution and Verification

To ensure quality, the framework uses a "rejection sampling" strategy. It verifies that an LLM cannot answer the question using internal knowledge alone, ensuring tool necessity. Furthermore, the system generates execution trajectories to provide a ground-truth path for training.

The execution flow of an agentic task within TaskCraft, illustrating the interaction between the LLM parser, tool execution, and reasoning steps.

The TaskCraft Dataset

The authors introduced the TaskCraft Dataset, a large-scale synthetic collection of approximately 36,000 tasks. The composition is diverse, with 75% web-based tasks, 15% image-based, and 10% PDF-based. These tasks cover critical domains including Finance, Academic Research, Government, and Culture. To optimize the generation process, the authors employed "bootstrap few-shot learning," which reduced generation time by nearly 20% while increasing the success rate to 68.1% for atomic tasks.

Key Results

  • Performance Boost: Fine-tuning a base model (Qwen2.5-3B) on TaskCraft's synthetic trajectories led to massive gains. On the HotpotQA benchmark, performance jumped from a baseline of 0.032 to 0.232—a 7x improvement—after Supervised Fine-Tuning (SFT).
  • Difficulty Alignment: The synthetic tasks successfully mirrored the difficulty levels of human-annotated benchmarks. Image-based tasks proved the most challenging, matching the failure rates seen in the most difficult levels of the GAIA benchmark.
  • Tool Necessity: TaskCraft explicitly filters for tasks where a tool-less LLM fails but a tool-equipped agent succeeds, ensuring the dataset specifically targets agentic capabilities.

Conclusion

TaskCraft demonstrates that high-quality, complex agentic data can be generated at scale without the need for exhaustive human annotation. By recursively expanding tasks and implementing rigorous verification, the framework provides a viable path for training the next generation of autonomous AI agents. Future work will focus on expanding the toolset to include specialized environments like databases and coding sandboxes.

Summary updated

Abstract

Agentic tasks, which require multi-step problem solving with autonomy, tool use, and adaptive reasoning, are becoming increasingly central to the advancement of NLP and AI. However, existing instruction data lacks tool interaction, and current agentic benchmarks rely on costly human annotation, limiting their scalability. We introduce \textsc{TaskCraft}, an automated workflow for generating difficulty-scalable, multi-tool, and verifiable agentic tasks with execution trajectories. TaskCraft expands atomic tasks using depth-based and width-based extensions to create structurally and hierarchically complex challenges. Empirical results show that these tasks improve prompt optimization in the generation workflow and enhance supervised fine-tuning of agentic foundation models. We present a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.
BibTeX
@inproceedings{shi2026taskcraft,
  author = {Dingfeng Shi and Jingyi Cao and Qianben Chen and Weichen Sun and Weizhen Li and Hongxuan Lu and Fangchen Dong and Tianrui Qin and King Zhu and Minghao Liu and Jian Yang and Ge Zhang and Jiaheng Liu and Changwang Zhang and Jun Wang and Yuchen Eleanor Jiang and Wangchunshu Zhou},
  title = {{TaskCraft: Automated Generation of Agentic Tasks}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=UJFCyrYM1V},
}
Figure