Introduction
The field of AI-driven software development has seen a surge in multi-agent systems (MAS) designed to automate the coding lifecycle. However, most existing frameworks, such as ChatDev, rely on rigid, "hard-coded" workflows—typically a linear Waterfall model (Design $\rightarrow$ Coding $\rightarrow$ Testing $\rightarrow$ Documentation). While this works for simple scripts, it fails to capture the fluid nature of real-world software engineering, where different tasks require bespoke processes.
A significant hurdle in moving toward dynamic workflows is "instance hallucination," where Large Language Models (LLMs) suggest illogical or non-existent development phases. The Think-on-Process (ToP) framework addresses this by enabling LLMs to dynamically generate, execute, and refine their own development processes through a closed-loop system.
The Think-on-Process (ToP) Framework
ToP introduces a four-stage iterative loop designed to move beyond static workflows and minimize hallucinations through experiential learning.
1. Instance Generating
In the first stage, the LLM acts as an "Instance Agent." Given a user request (e.g., "Develop a Tetris game"), it outlines a specific sequence of development phases tailored to that task. This allows for flexibility that a static model cannot provide.
2. Software Developing
Once a process instance is generated, a multi-agent system—comprising roles like CEO, CTO, Coder, and Tester—executes the task. The agents interact via a "Chat Chain," where an instructor agent guides an assistant agent through the specific steps defined in the instance.
3. Instance Filtering
To ensure quality, ToP employs a Replay mechanism based on the Upper Confidence Bound for Trees (UCT) algorithm. This stage filters out "hallucinated" or ineffective processes. A compiler acts as the ultimate judge; if the generated code fails to run, the associated process instance is discarded. Successful instances are stored in an "Instance Pool."
4. LLM Enhancing
The most innovative aspect of ToP is the use of Process Mining. By applying the Inductive Miner algorithm to the successful instances in the pool, the framework extracts a generalized "Process Model" (represented in BPMN). This model is then converted into a natural language description (BPM2Text) and fed back into the LLM's prompt. This "experiential knowledge" guides the LLM to generate more robust and logical processes in subsequent rounds.
Experimental Evaluation
The researchers evaluated ToP using the NLDD dataset, which contains over 1,200 software task prompts across categories like Games, Education, and Work.
Key Results
The ToP framework significantly outperformed baseline static models:
- Success Rate Boost: For GPT-3.5, the average success rate across all categories increased by 32%.
- Domain-Specific Gains: The "Game" category saw the most substantial improvement, with success rates climbing from 30% to 67%.
- High-End Performance: GPT-4, when guided by mined process descriptions, achieved success rates as high as 85%.
- Process Optimization: The framework reduced the complexity and "chaos" of generated workflows, cutting unnecessary decision points (gateways) by 65%.
Insights and Discussion
Several key factors contributed to ToP's success:
- Temperature Tuning: A "Goldilocks" temperature of 0.6 was found to be optimal, balancing the need for diverse process generation with the requirement for logical consistency.
- Environmental Feedback: Using a compiler as a ground-truth signal ensures the LLM learns from physical reality rather than just linguistic patterns.
- BPM2Text: Converting technical process diagrams into text proved to be a highly effective method for teaching complex logic to LLMs.
Conclusion
Think-on-Process represents a shift from "hard-coded" AI workflows to dynamic, self-improving systems. By integrating process mining with multi-agent collaboration, ToP allows LLMs to learn from their own successful experiences. Future work aims to incorporate negative constraints from failed instances and move toward dynamic role assignment, further mirroring the complexity of human software engineering teams.
Summary updated