WikiQA: A Challenge Dataset for Open-Domain Question Answering

Yi Yang, Wen-tau Yih, Christopher MeekEMNLP 2015 · OpenReview

Abstract

We describe the WIKIQA dataset, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering. Most previous work on answer sentence selection focuses on a dataset created using the TREC-QA data, which includes editor-generated questions and candidate answer sentences selected by matching content words in the question. WIKIQA is constructed using a more natural process and is more than an order of magnitude larger than the previous dataset. In addition, the WIKIQA dataset also includes questions for which there are no correct sentences, enabling researchers to work on answer triggering, a critical component in any QA system. We compare several systems on the task of answer sentence selection on both datasets and also describe the performance of a system on the problem of answer triggering using the WIKIQA dataset.

1. Introduction

The task of answer sentence selection—identifying the correct sentence from a candidate pool to answer a given question—is a cornerstone of open-domain question answering (QA). Historically, research in this area has been dominated by the QASENT (TREC-QA) dataset. However, QASENT suffers from significant lexical bias and the unrealistic assumption that an answer always exists within the candidate set. This paper introduces WikiQA, a large-scale, publicly available dataset designed to overcome these limitations by using real-world queries and introducing the challenge of "answer triggering."

2. The WikiQA Dataset

WikiQA represents a shift toward more naturalistic QA benchmarks. Unlike QASENT, which relied on editor-generated questions and keyword-matched candidates, WikiQA utilizes unedited queries from Bing search logs that led users to Wikipedia. For each question, the candidate pool consists of all sentences in the summary paragraph of the corresponding Wikipedia page.

A defining feature of WikiQA is its scale and the inclusion of questions without answers. As shown in Table 1, WikiQA contains over 3,000 questions, nearly two-thirds of which have no correct answer in the candidate set. This forces models to perform "answer triggering"—deciding if a valid answer exists at all—rather than just ranking candidates.

Table 1: Statistics of the WikiQA dataset, highlighting the inclusion of questions without answers.

3. Methodology

The authors evaluate several modeling approaches to establish baselines for the new dataset:

  • Baselines: Simple Word Count and Weighted Word Count (IDF).
  • Lexical Semantic Models: LCLR, which utilizes linguistic features like WordNet and lemma matching.
  • Deep Learning Models: Paragraph Vectors (PV) and Convolutional Neural Networks (CNN) to capture sentence-level semantics.
  • Hybrid Models: Combining CNN/PV scores with word-matching features (e.g., CNN-Cnt) via logistic regression.

4. Results and Analysis

The experimental results reveal a significant performance gap between the legacy QASENT dataset and the new WikiQA benchmark. On QASENT, lexical models like LCLR often perform comparably to or better than deep learning approaches due to the dataset's inherent keyword bias. However, on WikiQA, where 20.3% of correct answers share no content words with the question, the CNN-based models demonstrate clear superiority.

Table 2: Performance comparison of various models on QASENT and WikiQA datasets using MAP and MRR.

As seen in Table 2, the hybrid CNN-Cnt model achieves the highest performance on WikiQA with a MAP of 0.6520 and MRR of 0.6652. In contrast, the Word Count baseline's performance drops sharply on WikiQA compared to QASENT (from 0.5919 to 0.4891 MAP), confirming that WikiQA is a more rigorous test of semantic understanding.

5. Answer Triggering and Future Work

The answer triggering task remains a significant challenge. Even with additional features like question length (QLen), the best F1 score achieved was approximately 31.6%. This indicates that while current models are proficient at ranking, they struggle to determine the absolute presence of an answer.

Limitations include the models' inability to perform complex reasoning, such as inferring duration from date ranges (e.g., "How long was I Love Lucy on the air?"). Future research should focus on deeper semantic representations that account for temporal relationships and specific entity roles to build more reliable, production-ready QA systems.

Summary updated

Abstract

We describe the WIKIQA dataset, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering. Most previous work on answer sentence selection focuses on a dataset created using the TREC-QA data, which includes editor-generated questions and candidate answer sentences selected by matching content words in the question. WIKIQA is constructed using a more natural process and is more than an order of magnitude larger than the previous dataset. In addition, the WIKIQA dataset also includes questions for which there are no correct sentences, enabling researchers to work on answer triggering, a critical component in any QA system. We compare several systems on the task of answer sentence selection on both datasets and also describe the performance of a system on the problem of answer triggering using the WIKIQA dataset.
BibTeX
@inproceedings{yang2015wikiqa,
  author = {Yi Yang and Wen-tau Yih and Christopher Meek},
  title = {{WikiQA: A Challenge Dataset for Open-Domain Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=rk_wzGu-B},
}
Figure