Introduction
Standard Sequence-to-Sequence (Seq2Seq) models often fail when moving from synthetic "toy" datasets to the messy reality of human customer support. This paper introduces a hybrid approach that uses skip-connection Seq2Seq models to handle the logic of external actions (like API calls) and Nearest Neighbor retrieval to ensure responses remain fluent and human-like by "replaying" expert utterances.
The Challenge: Fluency vs. Coherency
Building task-oriented chatbots (agents that do things, like cancel a subscription) requires balancing two often-conflicting goals:
- Fluency: The response must sound like natural, grammatical English.
- Coherency: The response must make sense in the context of the dialogue and trigger the correct backend actions (API calls).
While state-of-the-art models perform beautifully on synthetic datasets like bAbI, they often fall apart on real-world data. In the wild, vocabulary is unconstrained and lexical variation is high. Purely generative models (Seq2Seq) often struggle to learn underlying grammar from limited data, leading to "coherent but ungrammatical" responses. Conversely, retrieval-based models are fluent but often lose the thread of the conversation or fail to trigger necessary system updates.
Proposed Architecture
The authors propose a two-pronged solution to bridge this gap:
- Skip-Connection Seq2Seq: To improve dialogue memory, the authors modify the standard architecture to pass context vectors from the final hidden units of both the encoder and decoder of the previous turn directly into the current turn. This "skip connection" ensures the model maintains a fixed-size representation of the entire dialogue history.
- Nearest Neighbor (NN) Retrieval: Instead of forcing the model to generate text word-by-word, they use the hidden state of the Seq2Seq model as an embedding. They then search the training data for the most similar dialogue state and "replay" the actual human agent's response from that state.
Datasets and Evaluation
The researchers evaluated their models on two primary benchmarks:
- bAbI (Tasks 1 & 2): A standard synthetic dataset involving simulated restaurant reservations.
- Customer Support (CS) Datasets: An internal Amazon dataset of 160k chat transcripts. This includes CS_large for training and CS_small, a subset of 1,000 transcripts hand-annotated by humans to mark exactly where API calls should occur.
Key Results
The hybrid "Best of Both Worlds" approach yielded significant improvements over baseline models:
- Massive Accuracy Boost: On the real-world dataset, the hybrid model achieved a 78% relative improvement in fluency (measured by BLEU) and a 200% improvement in external call accuracy (EQM) compared to baseline approaches.
- Skip-Connections Matter: On the synthetic bAbI dataset, adding skip-connections moved the Exact Query Match (EQM) from 0% to 55%, proving that standard Seq2Seq models struggle to track state across multiple turns without explicit architectural help.
- Human-in-the-Loop Validation: In "Online-BLEU" tests (where humans chose from the model's top 5 suggestions), the Nearest Neighbor approach using decoder outputs scored a staggering 90.8, compared to just 46.2 for the standard skip-connection model.
Conclusion and Future Work
By using hidden states as search queries and replaying expert utterances, the model can focus entirely on what to say rather than how to say it. While the model is currently limited by the static nature of its responses (it cannot generate entirely new sentences), it effectively addresses the massive "sim-to-real" gap in task-oriented dialogue systems. Future work will involve using Reinforcement Learning (RL) to help agents reach goal states in fewer turns.
Summary updated