Abstract
We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37,000 cloze-style queries constructed from definitions of software entity tags on Stack Overflow. The Quasar-T dataset consists of 43,000 open-domain trivia questions. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text, and (2) reading the retrieved text to answer the query. We evaluate several baselines and show that they lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively.
Introduction
In factoid Question Answering (QA), systems must navigate either structured Knowledge Bases or unstructured text. While recent Machine Reading Comprehension (MRC) models excel at extracting answers from single paragraphs, real-world QA requires a two-step pipeline: searching a massive corpus for relevant text and then reading that text to find the exact answer. Existing benchmarks often rely on proprietary search engines, hindering reproducible research on the retrieval step. The QUASAR datasets provide a standardized, large-scale benchmark for end-to-end search-and-read systems using controlled background corpora.
The QUASAR Datasets
The authors introduce two variants to test different domains:
- QUASAR-S (Software): Contains ~37,000 cloze-style questions about software entities. The background corpus consists of Stack Overflow posts, and answers are restricted to a fixed vocabulary of software tags.
- QUASAR-T (Trivia): Comprises ~43,000 open-domain trivia questions with free-form answers. It uses ClueWeb09 as its background corpus.
Unlike other benchmarks, QUASAR provides raw retrieved documents from standard corpora, allowing researchers to optimize the retrieval component directly.
Methodology
The datasets were constructed using automated scraping of Stack Overflow tags and Reddit trivia questions. For context retrieval, a two-phase system using Lucene/Indri indexes was implemented to fetch "pseudodocuments" (sentences or posts). Baseline evaluations included simple heuristics, language models (n-grams, BiRNN), and neural reading models like Gated-Attention (GA) Reader and BiDAF.
The Search-Read Tradeoff
A key insight from the study is the fundamental tension between search and reading accuracy. As the number of retrieved sentences or documents increases, the probability that the correct answer is included (Search Accuracy) rises. However, the increased volume of text introduces noise that degrades the reading model's ability to pinpoint the correct answer (Reading Accuracy).
Key Results
- Performance Gaps: On QUASAR-S, the best baseline (BiRNN) achieved 33.6% accuracy, compared to 50% for humans. On QUASAR-T, the BiDAF model reached a 28.5% F1 score, while humans achieved 60.6%.
- Retrieval Bottlenecks: Overall system performance is heavily limited by the initial retrieval phase. In QUASAR-T, search accuracy peaked at only ~40-45% with 20 documents, creating a hard ceiling for downstream reading models.
Conclusion
The QUASAR datasets highlight the significant gap between current neural models and human performance in open-domain QA. The results suggest that future progress requires moving beyond isolated pipelines toward joint, end-to-end training where retrieval is optimized specifically for the reading component's success.
Summary updated