Production Ready Chatbots: Generate if Not Retrieve

Aniruddha Tammewar, Monik Pamecha, Chirag Jain, Apurva Nagvenkar, Krupal ModiWorkshops @ AAAI 2018

TL;DR

This paper introduces a hybrid chatbot system that combines a rule-based graph dialogue model with a neural generative model to improve response accuracy and robustness in real-world applications. By leveraging the precision of rule-based systems for common flows and the flexibility of neural models for handling deviations, the system achieves significantly better performance than either approach alone.

What Problem Does This Paper Address?

Building effective chatbots for real-world applications, especially those handling complex user interactions like scheduling reminders, presents a significant challenge. Traditional rule-based systems offer high precision and grammatical correctness but suffer from low recall, meaning they can't handle a wide variety of user inputs or deviations from expected conversational paths. On the other hand, neural generative models can produce more varied and contextually relevant responses, but they are prone to grammatical errors and can struggle with maintaining a coherent conversational flow. The gap lies in creating a system that combines the strengths of both approaches to achieve both robustness and flexibility, catering to a wider range of user queries and conversational nuances without sacrificing accuracy.

Methods

The core of this work is a hybrid system that integrates two distinct components:

  • Graph-Based Dialogue Model: This component uses a directed graph to represent conversational states and transitions. Each state has associated intents, user message templates, and predefined responses. This model excels at handling predictable, "ideal" chat flows with high precision and without grammatical errors. It's built using hand-crafted rules and state machines, allowing for quick adaptation to product changes without extensive retraining.
  • Neural Conversational Model: This is a sequence-to-sequence (seq2seq) model, adapted from existing research, that uses recurrent neural networks (RNNs) with an attention mechanism. It's trained on conversational data to generate responses word-by-word. This model is specifically employed to handle situations where the graph-based model fails, such as dealing with spelling mistakes, deviations from ideal flows, or out-of-domain queries. The neural model is trained to either generate a text response or suggest an action that transitions the conversation back to a state within the graph model.

The system operates in real-time: user queries are first processed by the graph model. If the graph model can't handle the query (e.g., due to a low similarity score or an unidentifiable state), control is passed to the neural model. The neural model then attempts to generate a response or an action. If the neural model emits an action, it's performed, and the system state is updated. Thresholding mechanisms are in place to prevent the system from sending unfavorable responses or to shift control back to a human agent if the neural model fails to guide the conversation effectively.

Datasets

The paper describes using a corpus of dyadic conversations from the Reminders domain. This data was collected from interactions between Haptik users and human agents or the graph-based model. The dataset includes rich metadata like user information, timestamps, and message types.

Preprocessing was a significant step, reducing the dataset from 10.8 million to 3 million messages. Key preprocessing steps included:

  • Removal of out-of-domain conversations.
  • Merging of consecutive reminder notifications.
  • Replacement of named entities (date, time, etc.) with placeholder tags.
  • Extraction of actions from messages.
  • Orthographic normalization (lowercase, punctuation removal).

The final dataset used for training the neural model consisted of 611K conversations, split into training (80%) and validation (20%) sets.

Key Insights and Ideas

  • Hybridization is Key for Production: The paper strongly advocates for a hybrid approach as the most viable path to production-ready chatbots, especially in closed domains. Combining rule-based precision with neural flexibility addresses the limitations of each individual method.
  • Neural Model as a "Fallback" and "Guide": The neural model isn't just a generator; it's designed to gracefully handle errors and guide the user back to the structured conversational flows managed by the graph model. This is crucial for maintaining user experience and task completion.
  • Adaptability of Graph Models: The graph-based system's ability to adapt quickly to product changes (new features, UI elements) with minimal data is a significant advantage for real-world deployment, where requirements evolve rapidly.
  • Importance of Preprocessing for Data Efficiency: The extensive preprocessing steps highlight how crucial data cleaning and feature engineering are for training effective neural models, especially when dealing with real-world, noisy conversational data.

Key Results

The hybrid system demonstrated significant improvements over the graph-based system alone:

  • End-to-End Automation (E2E) Score: Increased from 69.23% to 77.11% (a 7.9% increment).
  • Automated Response (AOR) Score: Increased from 88.21% to 93.81% (a 5.6% increment).
  • AOR-E2E Score: Decreased from 18.98% to 16.7%, indicating that more chats that were previously partially handled are now being completed end-to-end by the hybrid system.

These results suggest the hybrid system effectively handles issues like spelling mistakes, deviations from ideal chat-flows, code-mixed queries, and out-of-domain queries that previously caused the graph model to fail.

Limitations and Future Work

Limitations:

  • Domain Shift: The model struggles with users who initiate out-of-domain queries within a conversation, as these are not always filtered out by the domain classifier. The model can generate correct text responses but may fail to perform the correct actions.
  • Gender Dependency: The model does not exhibit consistency in using gendered language, as it was trained on data from both male and female assistants.
  • Temporal Dependency: The model lacks a sense of time, leading to greetings that are not contextually appropriate for the time of day.
  • Limited Vocabulary: The model's vocabulary is relatively small, which can lead to related but incorrect responses.
  • "Not Favorable" Responses: The system can generate responses like "Sorry I cannot help you with that," which are identified as "not favorable" and need to be restricted.

Future Work:

  • Temporal and Gender Features: Incorporating temporal and gender information as features during decoding could address the temporal and gender dependency issues.
  • Adapting Neural Models to Product Changes: Investigating ways to adapt the neural model to new features or API calls with limited effort, potentially by pre-defining a larger vocabulary buffer for new words.
  • Improving Vocabulary: Training word vectors for a wider vocabulary and exploring copying mechanisms (as seen in recent seq2seq models) to mitigate vocabulary limitations.
  • Handling Out-of-Domain Actions: Tagging out-of-domain actions during preprocessing to improve handling of domain shifts.

Summary updated

BibTeX
@inproceedings{tammewar2018production,
  author = {Aniruddha Tammewar and Monik Pamecha and Chirag Jain and Apurva Nagvenkar and Krupal Modi},
  title = {{Production Ready Chatbots: Generate if Not Retrieve}},
  booktitle = {Workshops @ AAAI},
  year = {2018},
}
Figure