TL;DR
The paper introduces Chat2Workflow, the first benchmark designed to evaluate how well large language models (LLMs) can generate executable, deployable visual workflows (like those on Dify or Coze) from natural language across multi-turn interactions. The authors find that while state-of-the-art LLMs excel at understanding high-level user intent, they struggle to generate structurally sound and functionally correct workflows, leaving a substantial gap for true industrial automation.
What Problem Does This Paper Address?
In real-world industrial AI deployments, structured "agentic workflows" (explicitly orchestrated graphs of tasks and tools) are heavily favored over flexible but unpredictable autonomous agents (like ReAct). Popular platforms like Dify and Coze allow developers to visually stitch together nodes (LLMs, code execution, web search) into Directed Acyclic Graphs (DAGs).
However, building these workflows is currently a manual, time-consuming, and error-prone engineering task. As user requirements change, developers must constantly rewrite prompts and rewire graph logic. Can LLMs automate this process? While existing benchmarks test abstract tool use or single-turn code generation, they fail to evaluate the multi-turn, structurally constrained, and platform-executable realities of industrial workflow engineering.
Methods
The authors formalize workflow generation as a multi-turn interaction process where an LLM must output a structured JSON file representing the workflow's nodes and edges based on natural language instructions. This JSON is then automatically converted into a deployable YAML file compatible with production platforms like Dify.
To push the boundaries of current models, the paper proposes an Error-Driven Agentic Baseline built on the OpenCode framework. Instead of simple zero-shot prompting, this baseline introduces:
- SKILL-based structured prompting to enforce task-critical guidelines.
- Dynamic Variable Summaries extracted from previous turns to combat context decay over long dialogues.
- An Execution Loop with Auto-Repair Modules: A 5-attempt retry mechanism that catches common errors (such as JSON parsing failures, topological sorting violations, or node selection mismatches) and feeds the error logs back to the model for self-correction.
Datasets
The authors introduce the Chat2Workflow benchmark, a high-quality dataset consisting of 27 complex tasks split into 79 multi-turn instructions, yielding 237 total test instances. It was curated by human annotators from real-world production configurations and covers six distinct business domains.
To rigorously evaluate the models, the benchmark uses two progressive metrics:
- Pass Rate (%Pas.): Measures syntactic and structural validity (Does it compile into a valid YAML file? Are the variables consistent?).
- Resolve Rate (%Res.): Measures actual execution correctness (Does the workflow run without errors and produce the correct output when fed real test cases?).
Key Insights and Ideas
- Syntax $\neq$ Execution: A massive gap exists between a model's ability to generate a syntactically valid graph and a functionally working workflow. For instance, GLM-4.6 achieves a decent pass rate but suffers a massive drop in resolve rate, proving that standard formatting compliance does not guarantee problem-solving capability.
- The Multi-Turn Degradation Trap: As dialogue rounds progress and requirements evolve, workflow quality steadily declines for almost all models. Handling cumulative modifications without breaking existing graph logic remains a major bottleneck for LLMs.
- Reasoning Trumps Scale: Models explicitly optimized for reasoning (like Kimi-K2-Thinking) consistently outperform standard instruction-tuned models, even those with significantly larger parameter counts (like Qwen-3-Coder-480B).
Key Results
The benchmark evaluated 15 prominent models (4 closed-source, 11 open-source):
- Top Performer: Gemini-3-Pro-Preview achieved the highest absolute resolve rate at 60.20%, followed closely by Claude-Sonnet-4.5 and GPT-5.2.
- Open-Source Leader: GLM-4.7 led the open-source category with a 46.55% resolve rate, though it showed notable weaknesses in developer-centric tasks.
- Baseline Boost: The proposed Error-Driven Agentic Baseline successfully mitigated common structural errors, boosting the absolute resolve rate of GPT-5.1 by 4.78% and GPT-5.2 by 6.05%.
Limitations and Future Work
While Chat2Workflow represents a vital step forward, the authors note a few limitations:
- Vocabulary Scale: The benchmark simplifies node interfaces to 20 high-frequency node types to prioritize executability, whereas real-world platforms feature hundreds of community extensions and intricate parameter configurations.
- Dataset Scale: With 237 instances, the dataset is highly verified but may not capture the near-infinite variety of complex logic found in bespoke industrial business processes.
Future work will likely focus on expanding the node vocabulary and developing more sophisticated multi-turn memory mechanisms to prevent performance degradation over long-horizon tasks.
Summary updated