Abstract
Modern enterprise automation is transitioning from rigid, predefined processes to adaptive workflows where autonomous agents (APIs) are dynamically sequenced. A significant challenge in this transition is the "cold-start" problem, where the lack of historical execution data prevents traditional recommendation engines from identifying the "Next Best Agent" (NBA). This paper presents a multi-stage prompting framework that leverages Large Language Models (LLMs) to recommend agents based on semantic metadata. By decomposing the recommendation process into task-to-agent mapping, context-aware ranking, and data flow compatibility checks, the system effectively manages large agent repositories and provides accurate, zero-shot recommendations.
Introduction
In the evolving landscape of enterprise automation, "adaptive workflows" are replacing hard-coded business processes. These systems use autonomous agents to stitch together tasks dynamically based on user intent. However, recommending the "Next Best Agent" (NBA) is difficult when new agents are frequently added and historical data is unavailable—a classic cold-start problem.
Researchers from IBM Research AI, in a paper presented at AAAI-24, propose a structured, multi-stage prompting strategy using Large Language Models (LLMs) to overcome these hurdles.
The Cold-Start Challenge in Workflow Automation
Traditional business process management (BPM) relies on prescriptive workflows. While effective for stable processes, modern business needs—such as insurance claims or hospital admissions—require high flexibility. Current recommendation engines typically use historical logs (process mining) to suggest the next step. When an agent is new or the workflow is novel, these systems fail. Furthermore, the sheer scale of enterprise environments, often containing thousands of agents, exceeds the context window limits of most LLMs.
Methods: A Three-Stage Pipeline
To address scale and accuracy, the proposed framework utilizes a three-stage pipeline to narrow the search space and refine recommendations:
- Stage 1: Task-to-Agent Mapping (Offline): To handle thousands of agents, the system first identifies a subset capable of fulfilling a specific task. The LLM performs binary classification, matching agent metadata against task descriptions.
- Stage 2: Task-Specific NBAs (Online): Upon a user request, the system identifies the current "target agent" and "target task." It then prompts the LLM to rank the top-$k$ agents from the filtered list that logically follow the current one.
- Stage 3: Agent-Specific NBA (Online - Optional): This stage ensures data compatibility by checking if the output signature of the current agent semantically matches the input signature of potential successors.
To prevent hallucinations, the authors employed the Guidance library for controlled generation and used Sentence Transformers to map LLM outputs back to the validated agent database.
Evaluation and Key Results
The framework was evaluated on two datasets: the Action Based Conversational Dataset (ABCD) and a proprietary IBM Recruitment Process (RP) dataset.
- Superior Accuracy: On the ABCD dataset, the framework achieved an NBA accuracy of 51.25%, significantly outperforming semantic search (ChromaDB, 24.3%) and keyword search (Elastic Search, 18.3%).
- Zero-Shot Capability: The system successfully recommended agents without any historical training data, relying entirely on the semantic understanding of API names, descriptions, and signatures.
- Model Comparison: Testing across Flan-T5-XXL, OpenLlaMA-13B, and MPT-7B showed that smaller models like MPT-7B can be highly effective, particularly in matching data flow signatures.
- High Recall: Stage 1 achieved recall rates up to 91.2%, ensuring that relevant agents are rarely discarded during the initial filtering phase.
Conclusion and Future Work
The multi-stage prompting approach effectively navigates the cold-start problem by leveraging the semantic reasoning of LLMs. While the system is robust, its performance is tied to the quality of agent metadata. Future work includes integrating human-in-the-loop feedback to refine recommendations over time and deploying the framework within IBM Watson Orchestrate to support real-world enterprise users.
Summary updated