Abstract: In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.
Introduction
A major challenge in modern Natural Language Understanding (NLU) is the presence of "annotation artifacts." Many existing datasets rely on human annotators who are prompted to write questions based on specific texts. This often results in "easy" examples that models can solve using surface-level pattern matching or word-overlap heuristics rather than true reasoning.
To address this, the authors introduce BoolQ, a dataset of naturally occurring questions—queries written by real users in unconstrained settings (like Google Search) who do not yet know the answer. This approach captures the messy, non-factoid, and inferential nature of real-world curiosity.
The BoolQ Dataset
The researchers constructed BoolQ by mining anonymized Google search queries. They identified queries starting with indicator words (e.g., "did," "is," "can") and paired them with relevant Wikipedia passages. The resulting dataset contains 15,942 examples (9.4k train, 3.2k dev, 3.2k test), where each entry is a (Question, Passage, Answer) triplet.
The dataset covers a wide range of topics, from Entertainment and Science to Law and History. Notably, the questions are not merely factoid-based; they often involve existence, event occurrences, and definitional queries. While the dataset leans toward "Yes" answers (approximately 62%), the diversity of topics ensures that models cannot rely solely on simple heuristics.
Analyzing Reasoning Complexity
What makes BoolQ "surprisingly difficult" is the depth of inference required. Unlike prompted datasets where the answer is often a direct paraphrase of the text, BoolQ questions frequently require multi-step logic and world knowledge.
The authors categorized the reasoning required into several types. Only about 39% of questions can be solved by simple Paraphrasing. The remainder require more sophisticated logic:
- Factual Reasoning: Connecting passage statements to the question using world knowledge (e.g., knowing the White House is not in Toronto).
- Missing Mention: Inferring a "No" because a major event or entity would have been mentioned in the passage if it were true.
- Other Inference: Complex deductions, such as inferring that if one snake is the "most toxic," another cannot be.
Experimental Results and Transfer Learning
The study evaluated several state-of-the-art models, including BiLSTM with co-attention and massive pre-trained models like BERT-Large, OpenAI GPT, and ELMo. A key finding was the effectiveness of Transfer Learning.
The authors tested whether pre-training a model on a related task before fine-tuning it on BoolQ would improve performance. They found that transferring from Entailment datasets (like MultiNLI) was significantly more effective than transferring from extractive QA datasets (like SQuAD). This suggests that yes/no questions are functionally closer to Natural Language Inference (NLI) than to span-finding tasks.
Their best-performing method—a "pipeline" approach that trains BERT on MultiNLI before fine-tuning on BoolQ—achieved 80.4% accuracy. While impressive, this still lags significantly behind the 90% human accuracy baseline. This 10% gap highlights the remaining challenges in teaching models to handle natural, inferential queries.
Conclusion and Future Work
BoolQ demonstrates that naturally occurring questions are a robust test of a model's reasoning capabilities. While large-scale pre-training and NLI transfer provide substantial boosts, models still struggle with implicit and complex inference. Future work may involve expanding the task to require searching entire documents rather than single paragraphs, further testing the limits of modern NLU systems.
Summary updated