Quasar: Datasets for Question Answering by Search and Reading

Bhuwan Dhingra, Kathryn Mazaitis, William W. CohenarXiv 2017 · OpenReview

Abstract

We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37,000 cloze-style queries constructed from definitions of software entity tags on Stack Overflow. The Quasar-T dataset consists of 43,000 open-domain trivia questions. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text, and (2) reading the retrieved text to answer the query. We evaluate several baselines and show that they lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively.

Introduction

In factoid Question Answering (QA), systems must navigate either structured Knowledge Bases or unstructured text. While recent Machine Reading Comprehension (MRC) models excel at extracting answers from single paragraphs, real-world QA requires a two-step pipeline: searching a massive corpus for relevant text and then reading that text to find the exact answer. Existing benchmarks often rely on proprietary search engines, hindering reproducible research on the retrieval step. The QUASAR datasets provide a standardized, large-scale benchmark for end-to-end search-and-read systems using controlled background corpora.

The QUASAR Datasets

The authors introduce two variants to test different domains:

  • QUASAR-S (Software): Contains ~37,000 cloze-style questions about software entities. The background corpus consists of Stack Overflow posts, and answers are restricted to a fixed vocabulary of software tags.
  • QUASAR-T (Trivia): Comprises ~43,000 open-domain trivia questions with free-form answers. It uses ClueWeb09 as its background corpus.
Distribution of genres in the QUASAR-T dataset, showing a wide range of topics from people & places to sports.

Unlike other benchmarks, QUASAR provides raw retrieved documents from standard corpora, allowing researchers to optimize the retrieval component directly.

Methodology

The datasets were constructed using automated scraping of Stack Overflow tags and Reddit trivia questions. For context retrieval, a two-phase system using Lucene/Indri indexes was implemented to fetch "pseudodocuments" (sentences or posts). Baseline evaluations included simple heuristics, language models (n-grams, BiRNN), and neural reading models like Gated-Attention (GA) Reader and BiDAF.

The Search-Read Tradeoff

A key insight from the study is the fundamental tension between search and reading accuracy. As the number of retrieved sentences or documents increases, the probability that the correct answer is included (Search Accuracy) rises. However, the increased volume of text introduces noise that degrades the reading model's ability to pinpoint the correct answer (Reading Accuracy).

The tradeoff between search and reading accuracy on QUASAR-S: as more sentences are added to the context, search accuracy improves but reading accuracy declines.

Key Results

  • Performance Gaps: On QUASAR-S, the best baseline (BiRNN) achieved 33.6% accuracy, compared to 50% for humans. On QUASAR-T, the BiDAF model reached a 28.5% F1 score, while humans achieved 60.6%.
  • Retrieval Bottlenecks: Overall system performance is heavily limited by the initial retrieval phase. In QUASAR-T, search accuracy peaked at only ~40-45% with 20 documents, creating a hard ceiling for downstream reading models.

Conclusion

The QUASAR datasets highlight the significant gap between current neural models and human performance in open-domain QA. The results suggest that future progress requires moving beyond isolated pipelines toward joint, end-to-end training where retrieval is optimized specifically for the reading component's success.

Summary updated

Abstract

We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37000 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The Quasar-T dataset consists of 43000 open-domain trivia questions and their answers obtained from various internet sources. ClueWeb09 serves as the background corpus for extracting these answers. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text that include the correct answer to a query, and (2) reading the retrieved text to answer the query. We also describe a retrieval system for extracting relevant sentences and documents from the corpus given a query, and include these in the release for researchers wishing to only focus on (2). We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and show that these lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at https://github.com/bdhingra/quasar .
BibTeX
@misc{dhingra2017quasar,
  author = {Bhuwan Dhingra and Kathryn Mazaitis and William W. Cohen},
  title = {{Quasar: Datasets for Question Answering by Search and Reading}},
  year = {2017},
  eprint = {1707.03904},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.03904},
}
Figure