A Repository of Conversational Datasets

Matthew Henderson, Paweł Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, Tsung-Hsien Wen

TL;DR

This paper introduces a public repository of large-scale conversational datasets and a standardized evaluation framework for response selection models. By providing readily available data and consistent metrics, the authors aim to accelerate research and development in conversational AI, enabling more direct comparisons between different approaches.

What Problem Does This Paper Address?

Developing effective conversational AI systems, particularly those that can understand and respond naturally in dialogue, is heavily reliant on large, high-quality datasets. However, creating such datasets is a significant challenge. It often requires defining specific ontologies for task-oriented systems and meticulous manual annotation, leading to datasets that are scarce, limited in scope, and difficult to reproduce. This scarcity hinders progress in the field, making it hard to train robust models and compare different research efforts fairly. The authors aim to alleviate this by providing a centralized resource of diverse and massive conversational datasets.

Methods

The core contribution is a repository of three large conversational datasets: Reddit, OpenSubtitles, and AmazonQA. These datasets are processed and made available in a standardized format (Tensorflow record files) with consistent filtering and preprocessing steps. The repository also includes scripts to reproduce these datasets, allowing researchers to adapt the data preparation pipeline.

For evaluating conversational response selection models, the authors propose a standardized metric: 1-of-100 accuracy. This metric involves sampling 99 negative responses for each true response and measuring how often the correct response is ranked among the top 100. This approach offers a more direct and reproducible evaluation compared to generative model metrics like perplexity.

The paper also presents several baseline models for response selection, ranging from simple keyword-based methods (TF-IDF, BM25) to more sophisticated vector-based approaches using pre-trained embeddings (USE, USE-LARGE, ELMO, BERT-SMALL, BERT-LARGE) with similarity (SIM) and learned mapping (MAP) strategies. Finally, they introduce a competitive neural encoder model (POLYAI-ENCODER) trained on the entire dataset.

Datasets

The paper introduces a repository containing three large conversational datasets:

  • Reddit: Derived from billions of comments across millions of threads, offering a diverse range of discussions.
  • OpenSubtitles: Extracted from movie and television subtitles, providing natural conversational structures.
  • AmazonQA: Composed of question-answer pairs from Amazon product pages, focusing on e-commerce interactions.

These datasets are significantly larger than previously used benchmarks, with the Reddit data alone containing hundreds of millions of context-response pairs. The repository provides scripts to generate these datasets, ensuring reproducibility.

Key Insights and Ideas

  • Data Scale Matters: The sheer size of the provided datasets (hundreds of millions of examples) is crucial for training robust conversational models, especially for general-domain understanding.
  • Standardization is Key: A unified evaluation framework (1-of-100 accuracy) is essential for comparing diverse response selection models, overcoming the challenges of evaluating generative models.
  • Pre-training Potential: The authors suggest that these large, general-domain datasets can be used for pre-training models, which can then be fine-tuned for specific tasks with smaller, task-specific datasets, mirroring successful strategies in NLP.
  • Vector Embeddings Outperform Keywords: Vector-based methods, particularly those leveraging large pre-trained models like BERT, generally outperform traditional keyword-based approaches, though keyword methods show strength on specific domains like AmazonQA.
  • Learned Mappings Improve Performance: Simple learned linear mappings (MAP) over pre-trained embeddings consistently boost performance compared to direct cosine similarity (SIM), highlighting the benefit of adapting embeddings to the task.

Key Results

The paper presents 1-of-100 accuracy results on the three standard datasets. The POLYAI-ENCODER model significantly outperforms all baseline methods, achieving 61.3% on Reddit, 30.6% on OpenSubtitles, and 84.2% on AmazonQA. Among the baseline methods, vector-based approaches generally perform better than keyword-based ones, with the MAP variants consistently outperforming their SIM counterparts. For instance, USE-LARGE-MAP achieves 47.7% on Reddit, while BERT-LARGE-MAP reaches 24.0%. Keyword-based methods like TF-IDF and BM25 show competitive results on AmazonQA (51.8% and 52.3% respectively), likely due to the domain's reliance on specific product terms.

Limitations and Future Work

The authors acknowledge that the presented baselines are designed for quick execution and may not represent the state-of-the-art. They also note that while the 1-of-100 accuracy metric is a useful proxy, it doesn't perfectly capture user satisfaction.

Future work includes:

  • Adding more datasets to the repository.
  • Developing more competitive baseline models.
  • Expanding the benchmark results.
  • Encouraging contributions from the research community.

Summary updated

Abstract

Progress in Machine Learning is often driven by the availability of large datasets, and consistent evaluation metrics for comparing modeling approaches. To this end, we present a repository of conversational datasets consisting of hundreds of millions of examples, and a standardised evaluation procedure for conversational response selection models using '1-of-100 accuracy'. The repository contains scripts that allow researchers to reproduce the standard datasets, or to adapt the pre-processing and data filtering steps to their needs. We introduce and evaluate several competitive baselines for conversational response selection, whose implementations are shared in the repository, as well as a neural encoder model that is trained on the entire training set.
BibTeX
@inproceedings{henderson2019repository,
  author = {Matthew Henderson and Paweł Budzianowski and Iñigo Casanueva and Sam Coope and Daniela Gerz and Girish Kumar and Nikola Mrkšić and Georgios Spithourakis and Pei-Hao Su and Ivan Vulić and Tsung-Hsien Wen},
  title = {{A Repository of Conversational Datasets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure