Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement

Wangyang Ying, Yanchi Liu, Xujiang Zhao, Wei Cheng, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, Haifeng Chen

Abstract

Automatically extracting workflows as procedural graphs from natural language is promising yet underexplored, demanding both structural validity and logical alignment. We present text2flow, a multi-agent framework that formulates procedural graph extraction as a multi-round reasoning process with dedicated structural and logical refinement. The framework iterates through three stages: (1) a graph extraction phase with the graph builder agent, (2) a structural feedback phase in which a simulation agent diagnoses and explains structural defects, and (3) a logical feedback phase in which a semantic agent aligns semantics between flow logic and linguistic cues in the source text. Experiments demonstrate that text2flow achieves substantial improvements in both structural correctness and logical consistency over strong baselines.

Introduction: The Challenge of Procedural Extraction

Extracting workflows—such as business processes, medical protocols, or software scripts—from natural language is a high-stakes task. These "procedural graphs" must be more than visually correct; they must be structurally valid (no dead ends or unreachable steps) and logically faithful to the source text (correctly identifying parallel steps, alternatives, or specific conditions).

While Large Language Models (LLMs) excel at identifying individual actions, they often struggle with the "big picture," producing "ill-formed" structures or misinterpreting subtle linguistic cues—for example, using an "Exclusive OR" (XOR) gateway when the text implies an "Inclusive OR." Current self-correction methods often fail because the model "hallucinates" a fix without validating the underlying logic.

Methods: The text2flow Framework

The text2flow framework treats graph extraction as an iterative, multi-round reasoning process involving three specialized agents:

  1. Graph Builder Agent: Generates an initial procedural graph (represented as triples) using few-shot prompting.
  2. Simulation Agent (Structural Feedback): Treats the graph as executable code, running 10,000 stochastic simulations to identify "dead nodes" or topological issues, translating these into natural language feedback.
  3. Semantic Agent (Logical Feedback): Extracts "logic spans" from the source text and compares them to the graph's gateways, flagging inconsistencies (e.g., "A and B" text vs. XOR gateway).
The text2flow framework architecture, showing the iterative loop between graph generation and multi-agent evaluation.

To manage feedback, the system uses an Evaluation Sandbox with budget-constrained selection (modeled as a knapsack problem). It prioritizes feedback based on marginal utility (frequency of error in simulations) and a "RepeatFailure" score for persistent errors.

Key Insights and Ideas

  • Grounded Feedback: Unlike standard self-correction, text2flow uses "grounded" feedback from simulation results, providing an objective truth that breaks the cycle of compounding hallucinations.
  • Structural-First Priority: Logical consistency is only meaningful if the graph is structurally sound; unreachable branches cannot be logically validated.
  • Iterative Sweet Spot: Performance gains typically peak after two iterations. Beyond this, computational costs (token usage) increase significantly with diminishing returns in accuracy.
F1-score improvement and inference time per sample across iteration epochs.

Key Results

Evaluated on the PAGED benchmark (3,394 documents), text2flow consistently achieved the highest F1-scores across five LLM backbones (GPT-4o, Llama 3.1, Gemma 3, Qwen 3, and Mistral 3.2). Notable improvements were seen in complex gateway types; for instance, on GPT-4o, the F1-score for OR gateways improved from 0.296 to 0.356. Furthermore, this multi-agent approach outperformed supervised fine-tuning (SFT) on Llama-3.1-8B in most categories, demonstrating the power of sophisticated reasoning loops over raw data training.

Limitations and Future Work

The iterative nature of text2flow introduces higher token costs and inference latency compared to one-pass extraction. Additionally, while effective on structured procedural texts, its performance on highly ambiguous or informal documents remains to be explored. Future work may involve integrating domain-specific simulators or adaptive reward functions to handle more complex, non-linear workflows.

Summary updated

Abstract

Automatically extracting workflows as procedural graphs from natural language is promising yet underexplored, demanding both structural validity and logical alignment. While recent large language models (LLMs) show potential for procedural graph extraction, they often produce ill-formed structures or misinterpret logical flows. We present \model{}, a multi-agent framework that formulates procedural graph extraction as a multi-round reasoning process with dedicated structural and logical refinement. The framework iterates through three stages: (1) a graph extraction phase with the graph builder agent, (2) a structural feedback phase in which a simulation agent diagnoses and explains structural defects, and (3) a logical feedback phase in which a semantic agent aligns semantics between flow logic and linguistic cues in the source text. Important feedback is prioritized and expressed in natural language, which is injected into subsequent prompts, enabling interpretable and controllable refinement. This modular design allows agents to target distinct error types without supervision or parameter updates. Experiments demonstrate that \model{} achieves substantial improvements in both structural correctness and logical consistency over strong baselines.
BibTeX
@misc{ying2026multi,
  author = {Wangyang Ying and Yanchi Liu and Xujiang Zhao and Wei Cheng and Zhengzhang Chen and Wenchao Yu and Yanjie Fu and Haifeng Chen},
  title = {{Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement}},
  year = {2026},
  eprint = {2601.19170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.19170},
}
Figure