Training Neural Response Selection for Task-Oriented Dialogue Systems

Matthew Henderson, Ivan Vulic, Daniela Gerz, Iñigo Casanueva, Pawel Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrksic, Pei-Hao Su

Abstract

Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, Pei-Hao Su. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

2019.

Introduction

Building effective task-oriented dialogue systems—such as banking assistants or retail help desks—has historically been a "high-effort, low-data" challenge. Developers typically face a trade-off between two suboptimal paths. Modular systems require hand-crafting complex ontologies for every new domain, a process that is labor-intensive and fails to scale. Conversely, generative "chatbot" models often produce grammatically correct but semantically vacuous responses like "I don't know" or "Thanks," which are unsuitable for specific tasks.

Response selection offers a promising middle ground. By identifying the best response from a pre-defined set of high-quality candidates, systems can maintain task relevance and quality. However, traditional retrieval models require massive amounts of in-domain data to learn task-specific nuances—data that is rarely available for new applications. This paper introduces a "pretrain then fine-tune" framework that leverages massive general-domain data to bootstrap high-performance task-oriented systems.

Methodology

The authors propose a two-step neural training procedure designed to bridge the gap between general conversational ability and domain-specific expertise.

1. Response Selection Pretraining

The model is first pretrained on a massive general-domain corpus consisting of 727 million Reddit conversation pairs. A dual-encoder architecture is employed: one encoder processes the input context (query), while the other processes candidate responses. These are mapped into a joint semantic space where the model learns to maximize the scaled cosine similarity between a context and its true response while minimizing it for random negatives.

The dual-encoder architecture utilizes a "transformer-lite" design with n-gram embeddings and self-attention.

The architecture uses a "transformer-lite" approach. It combines unigram and bigram embeddings with positional encoding and self-attention, followed by feed-forward layers utilizing the swish activation function. This design balances representational power with computational efficiency.

2. Target Domain Fine-Tuning

The pretrained model is adapted to specific tasks using two strategies:

  • FT-DIRECT: Standard fine-tuning on a small in-domain dataset.
  • FT-MIXED: In-batch mixing of Reddit data and in-domain data to prevent "catastrophic forgetting" of general language patterns.

Datasets and Evaluation

The framework was evaluated across six diverse domains: Reddit (pretraining), Ubuntu (technical chat), OpenSubtitles (media dialogue), AmazonQA (e-commerce), SemEval15 (lifestyle/tourist info), and a newly introduced BANKING dataset. The BANKING dataset consists of FAQ-style e-banking queries across 77 unique categories, providing a rigorous test for intent-based response selection.

Results and Discussion

The primary evaluation metric is $R_{100}@1$, representing the probability that the correct response is ranked first among 100 candidates.

Performance Gains

The fine-tuned model demonstrated massive gains in low-data regimes. On the BANKING dataset, the model achieved a score of 94.8%, significantly outperforming the non-fine-tuned version (27.3%) and a model trained solely on banking data (88.3%). Furthermore, the proposed method consistently outperformed general-purpose encoders like BERT and the Universal Sentence Encoder (USE). For instance, on the Ubuntu dataset, the fine-tuned model reached 38.7%, while BERT-Large-MAP achieved only 8.0%.

Qualitative Impact of Fine-Tuning

The impact of fine-tuning is most visible in the organization of the model's semantic space. Qualitative analysis using t-SNE visualizations reveals that fine-tuning transforms a messy, overlapping space into one with clear, tight clusters for specific intents.

t-SNE visualizations demonstrate how fine-tuning organizes the semantic space into distinct intent clusters.

Key Insights

  • Pretraining for Retrieval: Unlike BERT or GPT, which are pretrained for general language modeling, this model is pretrained specifically for the mapping between inputs and responses, making it inherently superior for retrieval tasks.
  • The Importance of N-grams: Ablation studies showed that including bigrams and self-attention provided a massive boost over unigram-only models, suggesting that local word order is critical for distinguishing user intent.
  • Data Synergy: A key finding is that even minimal in-domain data, when used to fine-tune a massive pretrained model, yields better results than training from scratch. General conversational knowledge acts as a foundation for rapid domain adaptation.

Conclusion and Future Work

This research demonstrates that a dual-encoder model pretrained on massive conversational data and fine-tuned on small, domain-specific sets can achieve state-of-the-art performance in task-oriented dialogue. Future work will explore the use of "hard negatives" to further improve precision, as well as methods to incorporate deeper dialogue history into the context encoder.

Summary updated

BibTeX
@inproceedings{henderson2019training,
  author = {Matthew Henderson and Ivan Vulic and Daniela Gerz and Iñigo Casanueva and Pawel Budzianowski and Sam Coope and Georgios Spithourakis and Tsung-Hsien Wen and Nikola Mrksic and Pei-Hao Su},
  title = {{Training Neural Response Selection for Task-Oriented Dialogue Systems}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=RO8NvOhhinJ},
}
Figure