Dialog Flow Induction for Constrainable LLM-Based Chatbots

Stuti Agrawal, Pranav Pillai, Nishi Uppuluri, Revanth Gangi Reddy, Sha Li, Gokhan Tur, Dilek Hakkani-Tur, Heng Ji

TL;DR

This paper introduces an unsupervised method to automatically generate domain-specific dialog flows, which act as constraints for Large Language Model (LLM)-based chatbots. By leveraging LLMs like GPT-4, the approach creates structured conversation paths that help keep chatbots within their intended domains, improving accuracy and relevance, especially in specialized fields.

What Problem Does This Paper Address?

LLM-powered chatbots are increasingly popular due to their human-like conversational abilities. However, their strong generalization capabilities can be a double-edged sword: while useful for broad conversations, it makes it challenging to keep them focused on specific domains like healthcare or legal advice. This can lead to inaccurate information or irrelevant responses. Manually crafting dialog flows (structured conversation paths) to constrain these chatbots is a labor-intensive and time-consuming process, limiting scalability and coverage. The paper aims to solve this by automating the creation of these crucial dialog flows.

Methods

The core of the paper is an unsupervised approach to induce dialog flows, offering two variants:

  1. Intrinsic Dialog Flow: This method is used when no domain-specific conversation data is available. It leverages the LLM's (GPT-4) intrinsic knowledge of conversational principles and the domain's name. The process involves:
  2. Initial Flow Generation: GPT-4 generates an initial dialog flow based on a prompt including the domain name and desired structure.
  3. Feedback and Updates: GPT-4 self-evaluates the generated flow against criteria like representativeness, coverage, and clarity, providing actionable feedback. This feedback is used to suggest edits, which are then applied via an automated script.
  4. Finalization: A clean-up stage ensures the flow is structurally correct, removing inconsistencies.
  5. Data-Guided Dialog Flow: When domain-specific conversations are available, this approach enhances the intrinsic flow. It uses actual conversation instances to ground the generated flow to real-world dialog dynamics. The process includes:
  6. Identifying Representative Examples: Utterances from domain conversations are clustered and labeled using SentenceBERT embeddings and GPT-4. Representative conversations are then selected.
  7. Flow Generation: The identified representative examples (action labels and sample conversations) are used to condition the GPT-4 prompt for flow generation, followed by the same feedback, update, and clean-up stages as the intrinsic method.

A hybrid approach is also proposed, merging the broad scope of intrinsic flows with the specific focus of data-guided flows to create a "data-guided flow."

Datasets

The experiments were conducted on task-oriented dialog datasets:

  • MetaLWoz: 24 domains were used.
  • MultiWOZ: 5 domains were used.

For the data-guided flow induction, 80% of the data from each domain was used for training (conditioning the prompt), and the remaining 20% was reserved for evaluating coverage.

Key Insights and Ideas

  • LLMs as Flow Generators: The paper demonstrates that LLMs like GPT-4 can be effectively prompted to not only generate conversational content but also to design the underlying structure (dialog flows) of those conversations.
  • Unsupervised Flow Induction is Feasible: The proposed unsupervised methods significantly reduce the manual effort typically required for dialog flow creation, making it more scalable.
  • Data-Guided Flows Outperform Intrinsic Flows: Leveraging actual conversation data (data-guided flows) leads to better domain coverage compared to relying solely on an LLM's intrinsic knowledge (intrinsic flows). This highlights the importance of real-world examples for grounding.
  • Hybrid Approach Benefits: Merging intrinsic and data-guided flows offers a balanced approach, combining the broad coverage of intrinsic flows with the domain-specific accuracy of data-guided flows.

Key Results

  • Human Evaluation: Data-guided flows showed improved domain coverage (93.0% vs. 90.7% for intrinsic) and comparable scores for conclusiveness (87.7% vs. 87.8%) and coherence (84.8% vs. 84.5%) compared to intrinsic flows.
  • Automatic Evaluation: Data-driven flows demonstrated better coverage of bot-to-bot transitions (e.g., 43.0% for MultiWOZ data-driven vs. 39.9% for MultiWOZ intrinsic).
  • Kappa Agreement: A fair level of agreement (kappa=0.32) was observed among human annotators, with moderate agreement on domain coverage (kappa=0.46).

Limitations and Future Work

  • Task-Oriented Focus: The current experimentation is limited to task-oriented dialogs, which represent a narrower spectrum of dialog flows. Applicability to more open-ended dialogues needs further investigation.
  • Unsupervised Clustering: The methodology relies on unsupervised clustering, and the authors suggest that incorporating datasets with slot values and user intents could further enhance dialog flow induction.
  • Chatbot Performance: The paper focuses on generating flows but does not extend to testing the actual performance of chatbots constrained by these generated schemas. This remains an area for future work.
  • Future Directions: The authors envision these dialog flows as a springboard for future interactive dialog systems that can maintain natural conversation flow within specific domains.

Summary updated

Abstract

LLM-driven dialog systems are used in a diverse set of applications, ranging from healthcare to customer service. However, given their generalization capability, it is difficult to ensure that these chatbots stay within the boundaries of the specialized domains, potentially resulting in inaccurate information and irrelevant responses. This paper introduces an unsupervised approach for automatically inducing domain-specific dialog flows that can be used to constrain LLM-based chatbots. We introduce two variants of dialog flow based on the availability of in-domain conversation instances. Through human and automatic evaluation over 24 dialog domains, we demonstrate that our high-quality data-guided dialog flows achieve better domain coverage, thereby overcoming the need for extensive manual crafting of such flows.
BibTeX
@inproceedings{agrawal2024dialog,
  author = {Stuti Agrawal and Pranav Pillai and Nishi Uppuluri and Revanth Gangi Reddy and Sha Li and Gokhan Tur and Dilek Hakkani-Tur and Heng Ji},
  title = {{Dialog Flow Induction for Constrainable LLM-Based Chatbots}},
  booktitle = {SIGDIAL},
  year = {2024},
  url = {https://openreview.net/forum?id=3BEH0TnVeb},
}
Figure