A Repository of Conversational Datasets

Matthew Henderson, Pawel Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrksic, Georgios Spithourakis, Pei-Hao Su, Ivan Vulic, Tsung-Hsien Wen

Abstract: Progress in Machine Learning is often driven by the availability of large datasets, and consistent evaluation metrics for comparing modeling approaches. To this end, we present a repository of conversational datasets consisting of hundreds of millions of examples, and a standardised evaluation procedure for conversational response selection models using '1-of-100 accuracy'. The repository contains scripts that allow researchers to reproduce the standard datasets, or to adapt the pre-processing and data filtering steps to their needs. We introduce and evaluate several competitive baselines for conversational response selection, whose implementations are shared in the repository, as well as a neural encoder model that is trained on the entire training set.

Introduction

Building modern dialogue systems—the "brains" behind virtual assistants and customer service bots—requires enormous amounts of data. Historically, researchers relied on task-oriented datasets like MultiWOZ or DSTC2. While high-quality, these datasets are relatively small (often just a few thousand turns) and highly domain-specific. Collecting and annotating this data is a significant bottleneck, requiring strict ontologies and expensive human labeling.

To bypass this, researchers have looked toward "found" data online, but existing sources are often restricted in length or overly technical. This research addresses the scarcity of large-scale, diverse, and naturally structured conversational data by providing a reproducible pipeline for massive online corpora.

The Conversational Datasets

The repository introduces three primary datasets, significantly larger than previous benchmarks, processed using a standardized framework involving Apache Beam and Google Cloud Dataflow.

  • Reddit (2015–2018): The largest dataset in the collection, yielding 727 million context-response pairs after filtering. It covers over 300,000 sub-forums, providing high diversity.
  • OpenSubtitles (2018): Contains 316 million pairs derived from movie and TV scripts. While noisy due to ambiguous turn-taking, it offers a wealth of social, informal dialogue.
  • AmazonQA: A specialized set of 3.7 million question-answer pairs from product pages, ideal for e-commerce applications.
Summary of the conversational datasets provided in the repository, showing training and testing sizes.

Evaluation Framework and Baselines

To compare models fairly, the authors propose a "response selection" task. Given a conversation context, a model must identify the correct response from a pool of 100 candidates (1 ground truth and 99 random negatives). This is measured as 1-of-100 accuracy (Recall@1).

Several baseline approaches were evaluated:

  • Keyword-based: Traditional Information Retrieval (IR) methods like TF-IDF and BM25.
  • Vector-based: Using pre-trained embeddings (BERT, ELMo, Universal Sentence Encoder) to map text into a shared space. The authors compare simple cosine similarity (SIM) against a learned linear mapping (MAP).
  • POLYAI-ENCODER: A custom dual-encoder neural network using unigram/bigram features and self-attention to create sophisticated embeddings for both context and response.

Key Results and Insights

The custom POLYAI-ENCODER was the top performer across all datasets, demonstrating that a dedicated dual-encoder architecture trained on the full dataset is superior to generic pre-trained models.

1-of-100 accuracy results across different models and datasets.

Analysis of Performance

  • The "MAP" Advantage: Learning a simple linear mapping on top of fixed, pre-trained embeddings (MAP) significantly outperforms using raw embeddings (SIM). This suggests that off-the-shelf embeddings require task-specific tuning to understand the relationship between a question and an answer.
  • Domain Sensitivity: Keyword-based methods (TF-IDF) performed surprisingly well on AmazonQA (51.8% accuracy) because product names are unique and informative. However, they struggled on Reddit and OpenSubtitles, where semantic meaning and conversational "vibe" are more critical than word overlap.
  • Difficulty of OpenSubtitles: Models performed worst on OpenSubtitles, highlighting the difficulty of modeling dialogue where turn-taking structure isn't explicitly labeled.

Conclusion and Future Work

While the 1-of-100 metric is a robust benchmark, it assumes random negatives are always "bad" responses, which may not always hold true. Furthermore, while the datasets contain multi-turn histories, the current benchmarks primarily utilize the immediate previous turn. Future work will focus on utilizing deeper context and transitioning from response selection to generative models that can craft original responses from scratch using these massive datasets.

Summary updated

Abstract

Progress in Machine Learning is often driven by the availability of large datasets, and consistent evaluation metrics for comparing modeling approaches. To this end, we present a repository of conversational datasets consisting of hundreds of millions of examples, and a standardised evaluation procedure for conversational response selection models using '1-of-100 accuracy'. The repository contains scripts that allow researchers to reproduce the standard datasets, or to adapt the pre-processing and data filtering steps to their needs. We introduce and evaluate several competitive baselines for conversational response selection, whose implementations are shared in the repository, as well as a neural encoder model that is trained on the entire training set.
BibTeX
@inproceedings{henderson2019repository,
  author = {Matthew Henderson and Pawel Budzianowski and Iñigo Casanueva and Sam Coope and Daniela Gerz and Girish Kumar and Nikola Mrksic and Georgios Spithourakis and Pei-Hao Su and Ivan Vulic and Tsung-Hsien Wen},
  title = {{A Repository of Conversational Datasets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure