Deploying a Retrieval based Response Model for Task Oriented Dialogues

Lahari Poddar, György Szarvas, Cheng Wang, Jorge Balazs, Pavel Danchenko, Patrick ErnstEMNLP 2022

TL;DR

This paper presents a practical, three-stage approach to building effective retrieval-based dialogue systems for task-oriented domains like customer service. They introduce a method for semi-automatically creating a large response pool, a novel neural architecture that incorporates dialogue context and business constraints, and a training strategy combining self-supervised learning with human-in-the-loop fine-tuning. This approach leads to a system that is efficient, adaptable, and capable of handling real-world customer interactions.

What Problem Does This Paper Address?

Task-oriented dialogue (TOD) systems are crucial for automating customer service, but deploying them in real-world settings presents significant challenges. Existing end-to-end systems, particularly generative ones, struggle with guaranteeing response quality, style, and privacy. Retrieval-based systems, which select responses from a pre-defined pool, offer more control but need to be highly adaptable to diverse customer needs and business constraints while efficiently handling a vast number of potential responses. The paper addresses the need for a robust, scalable, and adaptable retrieval-based TOD system that can be effectively deployed in industry.

Methods

The authors propose a three-step procedure:

  1. Semi-Automatic Response Pool Creation: They develop an algorithm to extract a large, representative set of response templates from unannotated historical conversations. This method prioritizes frequency and novelty to ensure good coverage of natural language dialogues without requiring manual annotation.
  2. Neural Architecture: They enhance the Poly-Encoder architecture. This model uses separate transformers to encode dialogue history, profile features (like order details and business constraints), and candidate responses. Cross-attention layers then learn correlations between these components, and a 2-layer MLP merges history and profile information to represent the complete dialogue context. This architecture allows for efficient caching of response representations.
  3. Two-Stage Training Strategy:
    • Self-Supervised Training (SST): An initial model is trained on a large volume of anonymized human-human conversations. The goal is to rank correct utterances higher than randomly sampled ones, learning from dialogue history and profile features.
  4. Supervised Fine-Tuning (SFT) with Human-in-the-Loop: The SST model is used in a human-in-the-loop setting. Human agents are presented with top-ranked responses and can accept, modify, or indicate failure. This process generates a smaller, high-quality labeled dataset for fine-tuning the model, ensuring it aligns with business policies and customer needs.

Datasets

  • Training Data: Large volumes of anonymized human-human conversations were used for self-supervised training.
  • Evaluation Data: The authors used internal conversational datasets from an e-commerce customer service domain for two intents:
  • Start-Return (SR): Customers initiating a return.
  • Return-Refund Status (RRS): Customers inquiring about existing returns or refunds.
  • The datasets were split into training, development, and testing sets.
  • Human-in-the-Loop Data: Approximately 67,136 samples were collected over 8 weeks for training and evaluation of the fine-tuned models.

Key Insights and Ideas

  • Template Pool Coverage: A large, data-driven template pool, even if semi-automatically generated, can achieve significant conversational coverage, approaching human expert performance.
  • Human-in-the-Loop is Crucial: While self-supervised learning provides a strong baseline, human-in-the-loop fine-tuning is essential for adapting the model to specific business constraints and ensuring high-quality, policy-compliant responses.
  • Domain Complexity Matters: The performance of dialogue systems varies significantly with domain complexity. More procedural tasks (like starting a return) are easier to automate than open-ended inquiries (like status updates on complex return issues).
  • Arbitrary Human Choices: When presented with multiple similar, good options, human agents' choices can appear arbitrary, potentially leading to performance drops if not managed carefully.
  • Features are Indispensable: The lack of crucial features (e.g., granular tracking information) can severely limit a model's ability to handle complex scenarios, highlighting the importance of rich contextual information.
  • Transfer Learning with PLMs: Leveraging pre-trained language models (like BERT) as encoders significantly improves data efficiency and performance, even with limited supervised data.

Key Results

  • Offline Performance: The proposed model achieved strong offline metrics (Recall@k, MRR) on both SR and RRS intents.
  • Online Performance:
    • The self-supervised model achieved TAR@1 above 70% for SR and around 50% for RRS.
  • After supervised fine-tuning, the SR model achieved an Automated Task Completion (ATC) rate of 48.3% in live deployment, approaching human performance in controlled settings.
  • Using pre-trained language models (BERT) further boosted performance, with ATC reaching 55.3% for SR.
  • Inference Efficiency: The model's architecture allows for linear scaling of inference latency with template set size, enabling real-time conversations even with thousands of templates.
  • Template Diversity: An exploration mechanism (Gumbel-Softmax) was shown to help discover less frequent templates, improving diversity without degrading overall performance.

Limitations and Future Work

  • RRS Domain: The RRS intent still requires further work for robust deployment, particularly in incorporating essential package tracking information.
  • Human Choices: The arbitrary nature of human choices among similar templates needs further investigation to optimize agent productivity and model performance.
  • Future Work: The authors plan to deploy the improved BERT model for the RRS intent and further explore its performance and efficiency. They also suggest continued development with pre-trained language models and enhancing the exploration capabilities of the ranking model.

Summary updated

BibTeX
@inproceedings{poddar2022deploying,
  author = {Lahari Poddar and György Szarvas and Cheng Wang and Jorge Balazs and Pavel Danchenko and Patrick Ernst},
  title = {{Deploying a Retrieval based Response Model for Task Oriented Dialogues}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure