Multi-Stage Prompting for Next Best Agent Recommendations in Adaptive Workflows

Prerna Agarwal, Harshit Dave, Jayachandu Bandlamudi, Renuka Sindhgatta, Kushal MukherjeeAAAI 2024 · DOI

Abstract

Modern enterprise automation is transitioning from rigid, predefined processes to adaptive workflows where autonomous agents (APIs) are dynamically sequenced. A significant challenge in this transition is the "cold-start" problem, where the lack of historical execution data prevents traditional recommendation engines from identifying the "Next Best Agent" (NBA). This paper presents a multi-stage prompting framework that leverages Large Language Models (LLMs) to recommend agents based on semantic metadata. By decomposing the recommendation process into task-to-agent mapping, context-aware ranking, and data flow compatibility checks, the system effectively manages large agent repositories and provides accurate, zero-shot recommendations.

Introduction

In the evolving landscape of enterprise automation, "adaptive workflows" are replacing hard-coded business processes. These systems use autonomous agents to stitch together tasks dynamically based on user intent. However, recommending the "Next Best Agent" (NBA) is difficult when new agents are frequently added and historical data is unavailable—a classic cold-start problem.

The multi-stage pipeline as seen in a conversational interface, showing the progression from task mapping to specific agent recommendations.

Researchers from IBM Research AI, in a paper presented at AAAI-24, propose a structured, multi-stage prompting strategy using Large Language Models (LLMs) to overcome these hurdles.

The Cold-Start Challenge in Workflow Automation

Traditional business process management (BPM) relies on prescriptive workflows. While effective for stable processes, modern business needs—such as insurance claims or hospital admissions—require high flexibility. Current recommendation engines typically use historical logs (process mining) to suggest the next step. When an agent is new or the workflow is novel, these systems fail. Furthermore, the sheer scale of enterprise environments, often containing thousands of agents, exceeds the context window limits of most LLMs.

Methods: A Three-Stage Pipeline

To address scale and accuracy, the proposed framework utilizes a three-stage pipeline to narrow the search space and refine recommendations:

System architecture illustrating the flow from OpenAPI specifications to LLM-driven multi-stage prompting and post-processing.
  1. Stage 1: Task-to-Agent Mapping (Offline): To handle thousands of agents, the system first identifies a subset capable of fulfilling a specific task. The LLM performs binary classification, matching agent metadata against task descriptions.
  2. Stage 2: Task-Specific NBAs (Online): Upon a user request, the system identifies the current "target agent" and "target task." It then prompts the LLM to rank the top-$k$ agents from the filtered list that logically follow the current one.
  3. Stage 3: Agent-Specific NBA (Online - Optional): This stage ensures data compatibility by checking if the output signature of the current agent semantically matches the input signature of potential successors.

To prevent hallucinations, the authors employed the Guidance library for controlled generation and used Sentence Transformers to map LLM outputs back to the validated agent database.

Evaluation and Key Results

The framework was evaluated on two datasets: the Action Based Conversational Dataset (ABCD) and a proprietary IBM Recruitment Process (RP) dataset.

  • Superior Accuracy: On the ABCD dataset, the framework achieved an NBA accuracy of 51.25%, significantly outperforming semantic search (ChromaDB, 24.3%) and keyword search (Elastic Search, 18.3%).
  • Zero-Shot Capability: The system successfully recommended agents without any historical training data, relying entirely on the semantic understanding of API names, descriptions, and signatures.
  • Model Comparison: Testing across Flan-T5-XXL, OpenLlaMA-13B, and MPT-7B showed that smaller models like MPT-7B can be highly effective, particularly in matching data flow signatures.
  • High Recall: Stage 1 achieved recall rates up to 91.2%, ensuring that relevant agents are rarely discarded during the initial filtering phase.

Conclusion and Future Work

The multi-stage prompting approach effectively navigates the cold-start problem by leveraging the semantic reasoning of LLMs. While the system is robust, its performance is tied to the quality of agent metadata. Future work includes integrating human-in-the-loop feedback to refine recommendations over time and deploying the framework within IBM Watson Orchestrate to support real-world enterprise users.

Summary updated

BibTeX
@inproceedings{agarwal2024multi,
  author = {Prerna Agarwal and Harshit Dave and Jayachandu Bandlamudi and Renuka Sindhgatta and Kushal Mukherjee},
  title = {{Multi-Stage Prompting for Next Best Agent Recommendations in Adaptive Workflows}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30319},
}
Figure