TL;DR
This paper introduces FLOW-BENCH, a novel dataset and framework for generating enterprise workflows from natural language. FLOW-GEN leverages Large Language Models (LLMs) and an intermediate Python representation to translate user intents into structured BPMN workflows, addressing the limitations of current LLMs in this domain.
What Problem Does This Paper Address?
Modern enterprises rely heavily on Business Process Automation (BPA) tools to standardize operations and improve efficiency. While these tools offer visual interfaces, they often require significant engineering effort for complex integrations and custom logic. Natural Language (NL) interfaces promise to simplify BPA authoring, but existing LLMs struggle to generate accurate, structured workflows like BPMN due to a lack of specialized training data and the inherent verbosity of BPMN. This paper tackles the challenge of enabling LLMs to reliably generate enterprise workflows from conversational NL instructions, bridging the gap between user intent and executable BPA artifacts.
Methods
The core of the proposed solution, FLOW-GEN, utilizes a two-stage approach:
- NL to Python Intermediate Representation (IR): FLOW-GEN first translates NL instructions into a Python IR. This is a key innovation because LLMs are generally more proficient at generating Python code than the verbose BPMN format. The Python IR acts as a more compact and semantically rich representation of the workflow logic.
- Python IR to BPMN: A deterministic module (PY2BPMN) then converts this Python IR into standard BPMN, ensuring compatibility with existing BPA tools.
For FLOW-BENCH, the dataset creation involved:
- Quality Control: Filtering and simplifying existing workflows from commercial platforms.
- Manual Labeling: Crafting NL utterances and corresponding Python IR/BPMN representations.
- Data Augmentation: Creating incremental build steps (add, delete, replace) to simulate realistic workflow development.
The system also incorporates sophisticated retrieval mechanisms to ground LLM outputs in actual API catalogs, minimizing hallucinations. This includes:
- Activity Retrievers: Using embeddings (Embeddings_Retriever, Activities_Search) to find relevant API calls based on semantic similarity.
- Demonstration Retrievers: Employing cross-encoders (CE_Retriever) to select effective few-shot examples for the LLM prompt.
Datasets
The paper introduces FLOW-BENCH, a new benchmark dataset comprising 101 high-quality, manually curated incremental build step tests. Each test includes:
- Prior Sequence: The workflow state before an update.
- Utterance: The natural language instruction for the update.
- Expected Sequence: The resulting workflow after the update.
The dataset covers various workflow patterns (linear, conditional, loops, user tasks) and is grounded in an API catalog of 546 endpoints from diverse enterprise applications (GitHub, Jira, Slack, Salesforce, etc.).
Key Insights and Ideas
- Python IR as a Bridge: Using Python as an intermediate representation is a highly effective strategy for leveraging LLMs' strengths in code generation while simplifying the complex BPMN format.
- Incremental Updates are Crucial: Real-world workflow development often involves iterative changes. FLOW-BENCH's focus on incremental build steps is vital for evaluating systems that support this dynamic process.
- Grounding is Key to Avoiding Hallucinations: Directly linking LLM outputs to an API catalog is essential for generating accurate and reliable workflows, preventing the LLM from inventing non-existent actions.
- Cross-Encoders for Demonstrations: While embedding-based retrieval is good for activities, cross-encoders offer superior performance for selecting relevant few-shot demonstrations, leading to better LLM guidance.
Key Results
- Activity Retrieval: The
Activities_Searchretriever withTopK=50achieved the best recall (0.9926) and exact match score (0.7723) with the lowest hallucination rate (0.0102) on the FLOW-BENCH dataset. - Demonstration Retrieval: The
CE_Retriever(cross-encoder) consistently outperformedTopKRetriever(bi-encoder) for demonstration selection, improving exact match scores by up to 4 points. Retrieving five demonstrations was found to be optimal. - Model Performance: Among the evaluated LLMs,
Mistral-largeshowed the best performance with an exact match of 0.83 (in-domain) and 0.79 (cross-domain), and high Syntax F1 scores.llama-3-1-8b-instructperformed the worst. - BPMN Verbosity: BPMN representations are, on average, 25 times longer than their Python IR equivalents in FLOW-BENCH.
Limitations and Future Work
- Intermediate Representation Expressiveness: The Python IR intentionally omits domain-specific BPMN concepts like swimlanes and roles to maintain compactness and leverage LLM strengths. Future work could explore extending the IR or integrating with existing BPMN editors for these features.
- Hallucination and Activity Grounding: While the system models ungrounded activities as user tasks, LLMs may still generate hallucinated activity names. Users need to review generated workflows to distinguish between intentional user tasks and actual hallucinations.
- Dataset Scale and Domain Coverage: FLOW-BENCH, with 101 test cases and 546 activities, covers common enterprise domains but is a limited sample of real-world complexity. Expanding the dataset to more domains and complex nested workflows is a key future direction.
- Evaluation Scope: The current evaluation focuses on technical accuracy (Python IR quality). Future work should include user studies to assess usability, the quality of naturally generated utterances, and the long-term maintainability of conversationally constructed workflows.
Summary updated