Abstract
We consider real world task-oriented dialog settings, where agents need to generate both fluent natural language responses and correct external actions like database queries and updates. We demonstrate that, when applied to customer support chat transcripts, Sequence to Sequence (Seq2Seq) models often generate short, incoherent and ungrammatical natural language responses that are dominated by words that occur with high frequency in the training data. These phenomena do not arise in synthetic datasets such as bAbI, where we show Seq2Seq models are nearly perfect. We develop techniques to learn embeddings that succinctly capture relevant information from the dialog history, and demonstrate that nearest neighbor based approaches in this learned neural embedding space generate more fluent responses. However, we see that these methods are not able to accurately predict when to execute an external action. We show how to combine nearest neighbor and Seq2Seq methods in a hybrid model, where nearest neighbor is used to generate fluent responses and Seq2Seq type models ensure dialog coherency and generate accurate external actions. We show that this approach is well suited for customer support scenarios, where agents' responses are typically script-driven, and correct external actions are critically important. The hybrid model on the customer support data achieves a 78% relative improvement in fluency scores, and a 130% improvement in accuracy of external calls.
Introduction
In task-oriented dialog, AI agents must bridge the gap between human-like communication and precise logical execution. While generative Sequence-to-Sequence (Seq2Seq) models excel on synthetic datasets like bAbI, they often struggle with the complexities of real-world customer support transcripts. In these settings, standard models frequently produce "safe" but ungrammatical and incoherent responses, failing to maintain the fluency required for professional interaction. This paper explores a hybrid architecture designed to maintain high fluency while ensuring the accuracy of external API actions.
Methods: The Hybrid Model
We propose a hybrid system that combines the strengths of generative modeling and retrieval-based methods. The core of our approach lies in three components:
- Hierarchical Embedding Learning: We use a Hierarchical Recurrent Encoder-Decoder (HRED) to represent the entire dialog history as a fixed-size "belief state" vector. This captures context more effectively than traditional flat encoders.
- Nearest Neighbor (NNB) Retrieval: Instead of generating text word-by-word, the system uses the learned embedding to search a database of training responses. By finding the most similar historical context via a Ball Tree search, the model retrieves a human-written response. This guarantees grammatical fluency.
- The Hybrid Switch: The model is trained to predict whether the next turn requires a natural language response or an external API call. If an API call is needed, the Seq2Seq generator produces the code; otherwise, the NNB module retrieves a response.
Datasets and Preprocessing
Our experiments utilized two primary datasets:
- bAbI Tasks: Synthetic restaurant reservation dialogs where Seq2Seq models achieve near-perfect accuracy.
- Customer Support Dataset: 1,000 real-world chat transcripts involving account issues. This data required extensive preprocessing, including de-identification and lexical normalization, to handle typos and sensitive information.
Results and Discussion
The hybrid model significantly outperformed standalone generative models on the real-world dataset. Quantitatively, the hybrid approach (Model 5) achieved a 78% relative improvement in BLEU scores over the Seq2Seq baseline (Model 2), as shown in the table below.
Qualitatively, the hybrid model avoids the "average response" trap of Seq2Seq models, which often produce short, repetitive utterances (averaging 4.2 words). By retrieving human responses, the hybrid model maintains a natural utterance length (averaging 11.6 words) and professional tone.
Conclusion
By combining retrieval for natural language and generation for logical actions, our hybrid model addresses the fluency and coherency challenges of real-world task-oriented dialog. While the system shows marked improvements in both text quality and action accuracy, future work will focus on reducing response redundancy and conducting live human-in-the-loop evaluations.
Summary updated