Abstract
Despite their popularity in the chatbot literature, retrieval-based models have had modest impact on task-oriented dialogue systems, with the main obstacle to their application being the low-data regime of most task-oriented dialogue tasks. Inspired by the recent success of pretraining in language modelling, we propose an effective method for deploying response selection in task-oriented dialogue. To train response selection models for task-oriented dialogue tasks, we propose a novel method which: 1) pretrains the response selection model on large general-domain conversational corpora; and then 2) fine-tunes the pretrained model for the target dialogue domain, relying only on the small in-domain dataset to capture the nuances of the given dialogue domain. Our evaluation on six diverse application domains, ranging from e-commerce to banking, demonstrates the effectiveness of the proposed training method.
Introduction
Building effective task-oriented dialogue systems, such as customer support bots, is challenging due to the "low-data regime." Unlike general-purpose chatbots that can leverage vast internet data, specialized bots (e.g., for banking or technical support) must understand niche terminology with very few training examples.
Historically, two main approaches have been used:
- Generative Models: These can produce novel responses but are prone to "hallucinations" or defaulting to generic, unhelpful phrases.
- Retrieval Models: These select the best response from a fixed set. While more controllable and informative, they typically require large amounts of in-domain data to learn domain-specific nuances.
This research bridges the gap by leveraging large-scale general-domain conversations to provide a linguistic foundation before fine-tuning on specific tasks.
Methodology
The proposed framework consists of a two-step training procedure: large-scale pretraining followed by domain-specific fine-tuning.
1. Response Selection Pretraining
The model is first trained on a massive corpus of over 727 million comment-response pairs from Reddit. It employs a dual-encoder architecture to map both the input query and candidate responses into a shared 512-dimensional semantic space.
The architecture combines unigram and bigram embeddings with positional encoding and self-attention. A key finding was the effectiveness of the swish activation function ($x \cdot \text{sigmoid}(x)$) over standard ReLU or tanh. Scoring is performed using scaled cosine similarity, which facilitates better thresholding for response selection.
2. Target Domain Fine-Tuning
After pretraining, the model is adapted to specific domains using two strategies:
- FT-DIRECT: Continued training of the Reddit-pretrained model on the target dataset.
- FT-MIXED: Training on a mixture of target data and Reddit data (typically a 3:1 ratio) to maintain general language capabilities while learning domain specifics.
Experimental Setup
The framework was evaluated across six diverse domains:
- REDDIT: The source dataset for pretraining.
- OPENSUBTITLES: General movie/TV dialogue.
- AMAZONQA: E-commerce Q&A.
- UBUNTU: Technical support.
- BANKING: A new FAQ-style dataset for e-banking with 77 intent categories.
- SEMEVAL15: Lifestyle and tourism information.
Results and Analysis
The two-step approach significantly outperformed baselines, including BM25 and state-of-the-art encoders like BERT, ELMo, and USE.
Quantitative Performance
The gains from fine-tuning were most pronounced in specialized domains. In the Banking domain, fine-tuning resulted in a +67.5% improvement in Recall@1 over the Reddit-only model. The Ubuntu domain saw a +32.5% improvement. Furthermore, the dual-encoder design enabled a 130x speedup in retrieval through pre-computed embeddings and approximate nearest neighbor search.
Qualitative Insights
The model's ability to distinguish between subtle intents is best visualized through t-SNE embeddings. After fine-tuning, the model forms much tighter and more distinct clusters for specific categories compared to general-purpose encoders.
Key Insights
- Mapping vs. Similarity: Unlike standard encoders that seek semantic similarity, this model learns the mapping from a question to a valid response.
- N-gram Importance: Ablation studies showed that combining n-grams with self-attention provides a substantial performance boost over simpler models.
- Data Efficiency: The model achieves significant gains even with small target datasets, demonstrating that Reddit pretraining provides a robust linguistic foundation.
Conclusion and Future Work
This research demonstrates that pretraining on large-scale conversational data followed by domain-specific fine-tuning is a highly effective strategy for task-oriented dialogue. Future work may explore "hard" negative sampling to further improve accuracy and the incorporation of deeper multi-turn context. Additionally, balancing the trade-off between task specialization and general language retention remains an area for optimization.
Summary updated