Building Large Machine Reading-Comprehension Datasets using Paragraph Vectors

Radu Soricut, Nan DingarXiv 2016 · OpenReview

TL;DR

This paper introduces a novel method for automatically generating large-scale machine comprehension (MC) datasets using paragraph vectors, leading to a new benchmark of 2 million examples. They also propose a hybrid neural network architecture that combines recurrent and feedforward networks, achieving state-of-the-art results on their new dataset.

What Problem Does This Paper Address?

The core challenge addressed is the creation of robust machine comprehension (MC) systems that can truly understand natural language. A significant bottleneck in this field has been the lack of large, high-quality datasets for training and evaluating these systems. Existing datasets, while valuable, are often limited in size or scope, hindering the development of more sophisticated, data-hungry neural network models. The authors aim to bridge this gap by providing a scalable method for dataset generation and a powerful model architecture capable of leveraging such large datasets.

Methods

The paper presents two main methodological contributions:

  • MC-CREATE Algorithm: This algorithm leverages paragraph vector models (specifically, the PV-DBOW model) to automatically generate MC datasets. Given a corpus of (title, article) pairs, it trains a paragraph vector model to find "decoy" titles that are similar to a given title but not the correct answer for its associated article. The similarity is measured using a combination of surface-level string matching (BLEU) and paragraph-vector similarity. This process allows for the creation of datasets with a true answer and several plausible but incorrect "decoy" answers, mimicking human comprehension tests.
  • Hybrid Seq2seq + FFNN Architecture: The authors propose a novel neural network architecture that combines the strengths of recurrent neural networks (RNNs) and feedforward neural networks (FFNNs). Specifically, it uses a sequence-to-sequence (Seq2seq) model (with Gated Recurrent Units - GRUs and attention) to encode the article and title, and then feeds the encoder and decoder outputs into a feedforward network for classification. This hybrid approach aims to capture both sequential dependencies in text (via RNNs) and discriminative power (via FFNNs).

Datasets

  • MCAFP Dataset: The primary contribution is a newly generated MC dataset, named MCAFP, derived from the English Gigaword corpus (specifically, the AFP portion). This dataset contains approximately 2 million examples. The authors chose to restrict to a single publisher (AFP) to mitigate issues of information overlap and ensure the "one-correct-answer" assumption.
  • Existing Datasets (for context): The paper also discusses existing MC datasets like MCTest and CNN/DailyMail to highlight the limitations of current benchmarks and motivate the need for larger datasets.

Key Insights and Ideas

  • Scalable Dataset Generation: The MC-CREATE algorithm demonstrates that it's feasible to automatically generate large-scale MC datasets with a controlled level of difficulty and plausibility of incorrect answers, significantly reducing the manual effort required.
  • Hybrid Architectures Excel: The combination of Seq2seq (for sequence modeling) and FFNN (for classification) proved to be highly effective, outperforming simpler FFNN models and baselines. This suggests that capturing both sequential context and discriminative features is crucial for MC.
  • Human Performance Ceiling: The MCAFP dataset allows for a more accurate estimation of human performance, which the authors place around 91% accuracy. This provides a clear target for future model development.
  • Dataset Difficulty Matters: The authors found that the difficulty of decoy examples can influence model performance, with some models benefiting from a mix of easy and hard decoys, while others perform best with a more curated set.
  • Tied Embeddings and Bilinear Attention are Beneficial: For the Seq2seq+FFNN model, tying embedding matrices and using bilinear attention mechanisms led to significant improvements in accuracy and reduced model complexity.

Key Results

  • Human Performance: Estimated at around 91% accuracy on the MCAFP dataset.
  • Baseline Performance:
    • Random Guess: ~20% accuracy.
  • BLEU: ~51% accuracy.
  • Paragraph-Vectors: ~17.4% accuracy.
  • Neural Network Performance:
    • FFNN (5-class classifier): 58.1% accuracy.
  • FFNN (2-class classifier + argmax): 72.5% accuracy.
  • Seq2seq+FFNN (Hybrid Model): 83.2% accuracy on the test set. This is the state-of-the-art result reported in the paper.

Limitations and Future Work

  • Human Performance Gap: Despite achieving state-of-the-art results, the proposed model's 83.2% accuracy is still significantly below the estimated human performance ceiling of 91%. This indicates substantial room for future improvements.
  • Dataset Generation Nuances: While MC-CREATE is effective, the paper notes that the size of the initial corpus (C) is crucial for creating challenging datasets. They also discuss the potential impact of decoy difficulty on model training.
  • Future Work: The authors suggest that their MC-CREATE algorithm is language-agnostic and can be extended to other languages. They also believe their hybrid Seq2seq+FFNN model has potential for other tasks involving both generation and discrimination. Further research could explore different hyper-parameter settings, attention mechanisms, and training strategies to close the gap with human performance.

Summary updated

Abstract

We present a dual contribution to the task of machine reading-comprehension: a technique for creating large-sized machine-comprehension (MC) datasets using paragraph-vector models; and a novel, hybrid neural-network architecture that combines the representation power of recurrent neural networks with the discriminative power of fully-connected multi-layered networks. We use the MC-dataset generation technique to build a dataset of around 2 million examples, for which we empirically determine the high-ceiling of human performance (around 91% accuracy), as well as the performance of a variety of computer models. Among all the models we have experimented with, our hybrid neural-network architecture achieves the highest performance (83.2% accuracy). The remaining gap to the human-performance ceiling provides enough room for future model improvements.
BibTeX
@misc{soricut2016building,
  author = {Radu Soricut and Nan Ding},
  title = {{Building Large Machine Reading-Comprehension Datasets using Paragraph Vectors}},
  year = {2016},
  eprint = {1612.04342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1612.04342},
}
Figure