TL;DR
This paper introduces a novel approach to automatically generate conversational artifacts for digital assistants that interact with APIs and Robotic Process Automations (RPAs). By leveraging Large Language Models (LLMs) during a "build" phase, the system significantly reduces the effort required to create these assistants, making automation more accessible to non-technical users and avoiding costly LLM inference during runtime.
What Problem Does This Paper Address?
The increasing adoption of AI-driven automation, particularly through APIs and RPAs, faces a significant barrier: accessibility for business users who often lack the technical expertise to interact with these systems directly. While conversational interfaces (chatbots, virtual assistants) offer a more intuitive way to access automation, building them traditionally requires substantial AI and Natural Language Processing (NLP) expertise. This involves complex steps like training intent classifiers, identifying necessary input slots, and processing API outputs. Furthermore, relying on LLMs during runtime for these tasks can be prohibitively expensive and introduce issues like uncertain outputs and hallucinations. This paper addresses the need for an efficient, cost-effective, and human-guided method to create these conversational interfaces for automation technologies, specifically by reducing the reliance on AI expertise and LLM inference during the "use" phase.
Methods
The core of this paper's contribution lies in its "build phase" approach, where LLMs are used to generate conversational artifacts that are then embedded into an agent-based framework. This framework, inspired by prior work on multi-agent orchestrators for RPA and chatbots, defines "skills" (atomic functions like understanding intent, filling slots, acting, responding, and recommending next agents) and "agents" (compositions of these skills).
The key LLM-driven generation steps are:
- Utterance Generation: LLMs are prompted with API operation descriptions to generate diverse natural language utterances. These utterances serve as training data for intent classifiers, significantly augmenting limited human-annotated data. The process includes outlier removal and diversity selection to ensure high-quality training samples.
- Question Generation: For API calls requiring specific inputs (slots), LLMs are used to generate natural language questions that prompt users for this information. This is achieved by providing the LLM with API descriptions and attribute details, along with few-shot examples.
- Next Best Agent (NBA) Recommendation: The system uses semantic search (via ChromaDB and sentence embeddings) on agent descriptions to identify probable candidate agents that could follow the current one in a conversational flow. This is further refined by matching output signatures of the current agent with input signatures of candidate agents.
Crucially, these generated artifacts are reviewed and updated by a "builder" persona, ensuring human oversight and trust, and are then deployed for use. This offline generation means that during the "use phase," the system relies on pre-trained models and does not require expensive LLM inference.
Datasets
The paper utilizes several datasets for evaluation:
- SGD (Schema Guided Dialogue): A large-scale dataset for dialogue state tracking and API invocation, used for evaluating utterance generation and intent classification.
- HR (Human Resource): An internal dataset created from HR APIs and usage data, used for evaluating intent classification and NBA recommendations.
- ABCD (Action Based Conversational): A dataset of human-to-human dialogues for task-oriented conversations, used for evaluating NBA recommendations.
The paper does not introduce new benchmark datasets but rather leverages existing ones to validate its LLM-driven generation techniques.
Key Insights and Ideas
- LLMs as "Build-Time" Assistants: The most significant insight is the strategic use of LLMs during the build phase rather than the use phase. This effectively decouples the conversational agent from real-time LLM dependencies, leading to cost savings and improved reliability.
- Democratizing Automation: By generating conversational artifacts with LLMs and incorporating a human-in-the-loop review process, the system dramatically lowers the barrier to entry for creating digital assistants for automation, empowering non-AI experts.
- Augmenting Training Data: LLM-generated utterances are shown to be highly effective in augmenting limited human-annotated data for intent classification, leading to improved accuracy.
- Semantic Search for Agent Discovery: Employing semantic search with sentence embeddings for NBA recommendations allows for more nuanced and contextually relevant suggestions compared to purely keyword-based approaches.
Key Results
- Intent Classification: LLM-generated utterances significantly improved intent classification accuracy and F1-scores compared to a baseline model that only used API descriptions, across both WA and XGBoost classifiers and on HR and SGD datasets. For example, WA with FLAN-T5-XXL achieved an accuracy of 0.71 on SGD, compared to 0.49 for the baseline.
- Question Generation: The paper reports QG-Accuracy (a measure of semantic similarity to ground truth questions) for various LLMs, with Llama-2-13b-Chat, FLAN-T5-XL, and FLAN-UL2 showing strong performance. Notably, FLAN-T5-XL (3B parameters) outperformed the larger FLAN-T5-XXL (11B parameters), suggesting that model size isn't always the sole determinant of performance for this task.
- Next Best Agent Recommendation: The proposed approach outperformed an Elastic Search baseline on the ABCD dataset (NBA Accuracy: 0.26 vs. 0.10) and performed comparably on the HR dataset.
- Build Effort Reduction: Internal analysis indicated a 35-minute reduction in build effort per skill, translating to a 77% reduction.
Limitations and Future Work
- Human-in-the-Loop Dependency: While the human-guided build process enhances trust, it still requires human effort for review and updates. The paper acknowledges that the build persona may or may not modify artifacts, impacting performance.
- LLM Hallucinations/Uncertainty: Although LLMs are used offline, the potential for hallucinations and uncertain outputs during generation is a known LLM issue, which the outlier removal and human review aim to mitigate.
- Future Work: The authors plan to:
- Utilize human-in-the-loop validation more extensively to enhance methods.
- Incorporate end-user feedback for model retraining.
- Improve NBA performance by leveraging process knowledge and usage data.
- Explore online learning capabilities within the Watson Orchestrate product for continuous improvement.
Summary updated