What we need to learn if we want to do and not just talk

Rashmi Gangadharaiah, Balakrishnan Narayanaswamy, Charles ElkanNAACL 2018 · DOI

Introduction

Standard Sequence-to-Sequence (Seq2Seq) models often fail when moving from synthetic "toy" datasets to the messy reality of human customer support. This paper introduces a hybrid approach that uses skip-connection Seq2Seq models to handle the logic of external actions (like API calls) and Nearest Neighbor retrieval to ensure responses remain fluent and human-like by "replaying" expert utterances.

The interaction loop between a user, a human agent, and the chatbot providing suggestions.

The Challenge: Fluency vs. Coherency

Building task-oriented chatbots (agents that do things, like cancel a subscription) requires balancing two often-conflicting goals:

  1. Fluency: The response must sound like natural, grammatical English.
  2. Coherency: The response must make sense in the context of the dialogue and trigger the correct backend actions (API calls).

While state-of-the-art models perform beautifully on synthetic datasets like bAbI, they often fall apart on real-world data. In the wild, vocabulary is unconstrained and lexical variation is high. Purely generative models (Seq2Seq) often struggle to learn underlying grammar from limited data, leading to "coherent but ungrammatical" responses. Conversely, retrieval-based models are fluent but often lose the thread of the conversation or fail to trigger necessary system updates.

Proposed Architecture

The authors propose a two-pronged solution to bridge this gap:

  • Skip-Connection Seq2Seq: To improve dialogue memory, the authors modify the standard architecture to pass context vectors from the final hidden units of both the encoder and decoder of the previous turn directly into the current turn. This "skip connection" ensures the model maintains a fixed-size representation of the entire dialogue history.
  • Nearest Neighbor (NN) Retrieval: Instead of forcing the model to generate text word-by-word, they use the hidden state of the Seq2Seq model as an embedding. They then search the training data for the most similar dialogue state and "replay" the actual human agent's response from that state.
Architecture of the hybrid model showing skip-connections and the nearest neighbor retrieval mechanism.

Datasets and Evaluation

The researchers evaluated their models on two primary benchmarks:

  1. bAbI (Tasks 1 & 2): A standard synthetic dataset involving simulated restaurant reservations.
  2. Customer Support (CS) Datasets: An internal Amazon dataset of 160k chat transcripts. This includes CS_large for training and CS_small, a subset of 1,000 transcripts hand-annotated by humans to mark exactly where API calls should occur.

Key Results

The hybrid "Best of Both Worlds" approach yielded significant improvements over baseline models:

  • Massive Accuracy Boost: On the real-world dataset, the hybrid model achieved a 78% relative improvement in fluency (measured by BLEU) and a 200% improvement in external call accuracy (EQM) compared to baseline approaches.
  • Skip-Connections Matter: On the synthetic bAbI dataset, adding skip-connections moved the Exact Query Match (EQM) from 0% to 55%, proving that standard Seq2Seq models struggle to track state across multiple turns without explicit architectural help.
  • Human-in-the-Loop Validation: In "Online-BLEU" tests (where humans chose from the model's top 5 suggestions), the Nearest Neighbor approach using decoder outputs scored a staggering 90.8, compared to just 46.2 for the standard skip-connection model.

Conclusion and Future Work

By using hidden states as search queries and replaying expert utterances, the model can focus entirely on what to say rather than how to say it. While the model is currently limited by the static nature of its responses (it cannot generate entirely new sentences), it effectively addresses the massive "sim-to-real" gap in task-oriented dialogue systems. Future work will involve using Reinforcement Learning (RL) to help agents reach goal states in fewer turns.

Summary updated

BibTeX
@inproceedings{gangadharaiah2018what,
  author = {Rashmi Gangadharaiah and Balakrishnan Narayanaswamy and Charles Elkan},
  title = {{What we need to learn if we want to do and not just talk}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-3004},
}
Figure