Abstract
Automatically extracting workflows as procedural graphs from natural language is promising yet underexplored, demanding both structural validity and logical alignment. We present text2flow, a multi-agent framework that formulates procedural graph extraction as a multi-round reasoning process with dedicated structural and logical refinement. The framework iterates through three stages: (1) a graph extraction phase with the graph builder agent, (2) a structural feedback phase in which a simulation agent diagnoses and explains structural defects, and (3) a logical feedback phase in which a semantic agent aligns semantics between flow logic and linguistic cues in the source text. Experiments demonstrate that text2flow achieves substantial improvements in both structural correctness and logical consistency over strong baselines.
Introduction: The Challenge of Procedural Extraction
Extracting workflows—such as business processes, medical protocols, or software scripts—from natural language is a high-stakes task. These "procedural graphs" must be more than visually correct; they must be structurally valid (no dead ends or unreachable steps) and logically faithful to the source text (correctly identifying parallel steps, alternatives, or specific conditions).
While Large Language Models (LLMs) excel at identifying individual actions, they often struggle with the "big picture," producing "ill-formed" structures or misinterpreting subtle linguistic cues—for example, using an "Exclusive OR" (XOR) gateway when the text implies an "Inclusive OR." Current self-correction methods often fail because the model "hallucinates" a fix without validating the underlying logic.
Methods: The text2flow Framework
The text2flow framework treats graph extraction as an iterative, multi-round reasoning process involving three specialized agents:
- Graph Builder Agent: Generates an initial procedural graph (represented as triples) using few-shot prompting.
- Simulation Agent (Structural Feedback): Treats the graph as executable code, running 10,000 stochastic simulations to identify "dead nodes" or topological issues, translating these into natural language feedback.
- Semantic Agent (Logical Feedback): Extracts "logic spans" from the source text and compares them to the graph's gateways, flagging inconsistencies (e.g., "A and B" text vs. XOR gateway).
To manage feedback, the system uses an Evaluation Sandbox with budget-constrained selection (modeled as a knapsack problem). It prioritizes feedback based on marginal utility (frequency of error in simulations) and a "RepeatFailure" score for persistent errors.
Key Insights and Ideas
- Grounded Feedback: Unlike standard self-correction, text2flow uses "grounded" feedback from simulation results, providing an objective truth that breaks the cycle of compounding hallucinations.
- Structural-First Priority: Logical consistency is only meaningful if the graph is structurally sound; unreachable branches cannot be logically validated.
- Iterative Sweet Spot: Performance gains typically peak after two iterations. Beyond this, computational costs (token usage) increase significantly with diminishing returns in accuracy.
Key Results
Evaluated on the PAGED benchmark (3,394 documents), text2flow consistently achieved the highest F1-scores across five LLM backbones (GPT-4o, Llama 3.1, Gemma 3, Qwen 3, and Mistral 3.2). Notable improvements were seen in complex gateway types; for instance, on GPT-4o, the F1-score for OR gateways improved from 0.296 to 0.356. Furthermore, this multi-agent approach outperformed supervised fine-tuning (SFT) on Llama-3.1-8B in most categories, demonstrating the power of sophisticated reasoning loops over raw data training.
Limitations and Future Work
The iterative nature of text2flow introduces higher token costs and inference latency compared to one-pass extraction. Additionally, while effective on structured procedural texts, its performance on highly ambiguous or informal documents remains to be explored. Future work may involve integrating domain-specific simulators or adaptive reward functions to handle more complex, non-linear workflows.
Summary updated