Bootstrapping Dialog Models from Human to Human Conversation Logs

Pankaj Dhoolia, Vineet Kumar, Danish Contractor, Sachindra JoshiAAAI 2021 · OpenReview

Abstract

Building conversational agents currently requires significant manual effort from dialog designers and domain experts to define intents and interaction flows. This process is often disconnected from how real users actually communicate. We present a research prototype that automates this by ingesting human-to-human conversation logs. By utilizing "user" and "agent" emulators, the system suggests data-driven intents and responses, enabling designers to bootstrap robust dialog models efficiently. To our knowledge, this is the first system that enables data-driven dialog model creation by emulating both users and agents.

The Bottleneck in Bot Design

State-of-the-art commercial platforms like IBM Watson Assistant offer powerful tools for bot creation, but the design phase remains a significant hurdle. Traditionally, a designer must manually map out every possible user intent and the corresponding logic tree (dialog flow). This manual approach is slow, expensive, and often results in "artificial" interactions that fail to account for the linguistic variety found in real-world support chat transcripts. Designers often have to guess how users will phrase things, leading to bots that break when faced with real-world variations.

The Proposed System: Emulating the Conversation

To bridge the gap between raw logs and structured models, the system acts as an interactive assistant for the bot designer. It processes historical conversation logs to create two distinct emulators that model the behavior of both participants in a dialogue.

Architecture of the data-driven dialog bootstrapping system.

The workflow involves two primary paths as shown in the architecture above:

  1. User Emulation: Given an "agent context," the user emulator searches logs for similar historical situations to suggest what a user might say next. These utterances are then clustered to generate potential intents for the Watson Assistant Skill.
  2. Agent Emulation: Conversely, given a "user context," the agent emulator suggests how a professional agent responded in the past. These responses are clustered to help update the dialog flow with realistic agent replies.

Technical Implementation

The system leverages several advanced techniques to ensure the quality and diversity of its suggestions:

  • Contextual Embeddings: It uses the Universal Sentence Encoder (USE) to map utterances into a high-dimensional vector space, allowing the system to recognize semantic similarity even when keywords differ (e.g., "I can't log in" vs. "My password isn't working").
  • Efficient Retrieval: The Annoy library is used for fast approximate nearest neighbor searches within the conversation logs to find relevant historical contexts.
  • Intent Generation: K-means clustering groups thousands of log entries into manageable, representative intents.
  • Diversity Sampling: To prevent repetitive suggestions, a Determinantal Point Process (DPP) is employed. This ensures the designer sees a diverse range of possible user behaviors rather than multiple variations of the same phrase.

Key Insights and Results

A critical finding during development was the "Shortest First" heuristic. Because sentence embeddings can degrade with length, processing shorter, more concise sentences first led to significantly cleaner and more coherent intent clusters.

The prototype demonstrates that data-driven grounding significantly improves bot coverage. By building from real phrasing, the resulting bots are more resilient to natural language variations. Furthermore, the integration into IBM Watson Assistant proves that this automated bootstrapping can be seamlessly incorporated into industry-standard workflows, drastically reducing the time and expertise required to move from raw data to a functional conversational agent.

Limitations and Future Outlook

The system's performance is inherently tied to the quality of the input logs. Messy or unhelpful human-to-human transcripts will yield poor suggestions. Additionally, while sentence encoders are powerful, they still struggle with extremely long or complex utterances. Future work may explore active learning, where the system continues to refine its models as it encounters new production data.

Summary updated

Abstract

State-of-the-art commercial dialog platforms provide powerful tools to build a conversational agent. These platforms provide complete control to the dialog designer to model user-agent interactions. However, a dialog designer needs to rely on domain experts to manually build the dialog model -- by creating dialog flow nodes and modeling user intents. This process is laborious, time consuming and expensive and does not allow the designer to exploit human to human conversation logs effectively. In this work, we present a research prototype that can ingest human-to-human conversation logs between an end-user and an agent, and suggest user-intents and agent-responses, given a conversation context. We utilize human to human conversation logs to build two emulators: user and agent. An agent emulator models an agent response given the conversation context so far, and a user emulator outputs possible user responses. Our system is able to recommend conversational intents as well as conversation flow using emulators based on real-world data, thus making the process of designing a bot more efficient. To the best our knowledge this is the first system that enables data-driven dialog model creation by emulating users and agents.
BibTeX
@inproceedings{dhoolia2021bootstrapping,
  author = {Pankaj Dhoolia and Vineet Kumar and Danish Contractor and Sachindra Joshi},
  title = {{Bootstrapping Dialog Models from Human to Human Conversation Logs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=tprhA0aVtFl},
}
Figure