Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language

Yi Zhong, Buqiang Xu, Yijun Wang, Zifei Shan, Shuofei Qiao, Guozhou Zheng, Ningyu ZhangarXiv 2026 · arXiv

TL;DR

The paper introduces Chat2Workflow, the first benchmark designed to evaluate how well large language models (LLMs) can generate executable, deployable visual workflows (like those on Dify or Coze) from natural language across multi-turn interactions. The authors find that while state-of-the-art LLMs excel at understanding high-level user intent, they struggle to generate structurally sound and functionally correct workflows, leaving a substantial gap for true industrial automation.

What Problem Does This Paper Address?

In real-world industrial AI deployments, structured "agentic workflows" (explicitly orchestrated graphs of tasks and tools) are heavily favored over flexible but unpredictable autonomous agents (like ReAct). Popular platforms like Dify and Coze allow developers to visually stitch together nodes (LLMs, code execution, web search) into Directed Acyclic Graphs (DAGs).

However, building these workflows is currently a manual, time-consuming, and error-prone engineering task. As user requirements change, developers must constantly rewrite prompts and rewire graph logic. Can LLMs automate this process? While existing benchmarks test abstract tool use or single-turn code generation, they fail to evaluate the multi-turn, structurally constrained, and platform-executable realities of industrial workflow engineering.

Methods

The authors formalize workflow generation as a multi-turn interaction process where an LLM must output a structured JSON file representing the workflow's nodes and edges based on natural language instructions. This JSON is then automatically converted into a deployable YAML file compatible with production platforms like Dify.

To push the boundaries of current models, the paper proposes an Error-Driven Agentic Baseline built on the OpenCode framework. Instead of simple zero-shot prompting, this baseline introduces:

  1. SKILL-based structured prompting to enforce task-critical guidelines.
  2. Dynamic Variable Summaries extracted from previous turns to combat context decay over long dialogues.
  3. An Execution Loop with Auto-Repair Modules: A 5-attempt retry mechanism that catches common errors (such as JSON parsing failures, topological sorting violations, or node selection mismatches) and feeds the error logs back to the model for self-correction.

Datasets

The authors introduce the Chat2Workflow benchmark, a high-quality dataset consisting of 27 complex tasks split into 79 multi-turn instructions, yielding 237 total test instances. It was curated by human annotators from real-world production configurations and covers six distinct business domains.

Figure 2: Distribution of task types in Chat2Workflow. The benchmark covers six domains: AIGC, Research, Document, Education, Enterprise, and Developer.

To rigorously evaluate the models, the benchmark uses two progressive metrics:

  • Pass Rate (%Pas.): Measures syntactic and structural validity (Does it compile into a valid YAML file? Are the variables consistent?).
  • Resolve Rate (%Res.): Measures actual execution correctness (Does the workflow run without errors and produce the correct output when fed real test cases?).

Key Insights and Ideas

  • Syntax $\neq$ Execution: A massive gap exists between a model's ability to generate a syntactically valid graph and a functionally working workflow. For instance, GLM-4.6 achieves a decent pass rate but suffers a massive drop in resolve rate, proving that standard formatting compliance does not guarantee problem-solving capability.
  • The Multi-Turn Degradation Trap: As dialogue rounds progress and requirements evolve, workflow quality steadily declines for almost all models. Handling cumulative modifications without breaking existing graph logic remains a major bottleneck for LLMs.
Figure 4: Performance degradation across dialogue rounds. We show the Pass Rate and Resolve Rate for all 15 models across the first three dialogue rounds. Most models exhibit a steady decline in both metrics as the number of interaction rounds increases, indicating the challenge of maintaining workflow quality under evolving requirements.
  • Reasoning Trumps Scale: Models explicitly optimized for reasoning (like Kimi-K2-Thinking) consistently outperform standard instruction-tuned models, even those with significantly larger parameter counts (like Qwen-3-Coder-480B).

Key Results

The benchmark evaluated 15 prominent models (4 closed-source, 11 open-source):

  • Top Performer: Gemini-3-Pro-Preview achieved the highest absolute resolve rate at 60.20%, followed closely by Claude-Sonnet-4.5 and GPT-5.2.
  • Open-Source Leader: GLM-4.7 led the open-source category with a 46.55% resolve rate, though it showed notable weaknesses in developer-centric tasks.
  • Baseline Boost: The proposed Error-Driven Agentic Baseline successfully mitigated common structural errors, boosting the absolute resolve rate of GPT-5.1 by 4.78% and GPT-5.2 by 6.05%.
Figure 7: Performance comparison demonstrating the improvements of the proposed Agentic baseline over the Zero-Shot setting for GPT-5.1 and GPT-5.2.

Limitations and Future Work

While Chat2Workflow represents a vital step forward, the authors note a few limitations:

  1. Vocabulary Scale: The benchmark simplifies node interfaces to 20 high-frequency node types to prioritize executability, whereas real-world platforms feature hundreds of community extensions and intricate parameter configurations.
  2. Dataset Scale: With 237 instances, the dataset is highly verified but may not capture the near-infinite variety of complex logic found in bespoke industrial business processes.

Future work will likely focus on expanding the node vocabulary and developing more sophisticated multi-turn memory mechanisms to prevent performance degradation over long-horizon tasks.

Summary updated

Abstract

At present, executable visual workflows have emerged as a mainstream paradigm in real-world industrial deployments, offering strong reliability and controllability. However, in current practice, such workflows are almost entirely constructed through manual engineering: developers must carefully design workflows, write prompts for each step, and repeatedly revise the logic as requirements evolve -- making development costly, time-consuming, and error-prone. To study whether large language models can automate this multi-round interaction process, we introduce Chat2Workflow, a benchmark for generating executable visual workflows directly from natural language, and propose a robust agentic baseline to improve performance. The benchmark is built from a large collection of real-world business workflows, with each instance designed so that the generated workflow can be transformed and directly deployed to practical workflow platforms such as Dify and Coze. Experimental results show that while state-of-the-art language models can often capture high-level intent, they struggle to generate correct, stable, and executable workflows, especially given complex and evolving requirements. Although our agentic baseline yields up to 6.05% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation. Code is available at https://github.com/zjunlp/Chat2Workflow.
BibTeX
@misc{zhong2026chat,
  author = {Yi Zhong and Buqiang Xu and Yijun Wang and Zifei Shan and Shuofei Qiao and Guozhou Zheng and Ningyu Zhang},
  title = {{Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language}},
  year = {2026},
  eprint = {2604.19667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.19667},
}

Loading related content...

Figure