Abstract
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
2019.
Introduction: Beyond Factoid QA
In Question Answering (QA), benchmarks often focus on "factoid" queries (e.g., "What is the capital of France?") where the answer is a specific entity. Many existing datasets are constructed by showing crowd-workers a text and asking them to generate questions. This often leads to "surface-level" questions that models can solve through simple keyword matching rather than deep linguistic understanding.
The BoolQ paper introduces a dataset of naturally occurring yes/no questions—queries asked by users in search engines who do not already know the answer and were not prompted by a specific text. These questions are fundamentally different: they often query non-factoid information and require complex, entailment-like inference. For example, answering "Does France have a Prime Minister and a President?" requires a model to infer the existence of both roles from a passage that might only discuss their respective powers, rather than finding a single sentence stating both exist.
The BoolQ Dataset and Reasoning Diversity
The BoolQ dataset consists of 15,942 examples. Questions were harvested from anonymized Google search queries, and passages were sourced from Wikipedia. A key insight is that people tend to ask yes/no questions specifically when the answer is complex; if the answer were a simple fact, they would likely use a "who/what/where" format. This creates a natural dataset of difficult reasoning problems.
The reasoning required for BoolQ is highly diverse. The authors categorized these into types such as "Missing Mention" (inferring a 'No' because an event wasn't mentioned where it should have been) and "By Example" (inferring a 'Yes' because the text provides a specific instance). Notably, only 38.7% of BoolQ can be solved by simple paraphrasing, highlighting the dataset's inherent difficulty compared to traditional benchmarks.
Methodology: The Power of Transfer Learning
To address these challenges, the researchers explored several modeling strategies, moving from shallow baselines to massive pre-trained Transformers like OpenAI GPT and BERT-Large. The core strategy involved multi-step transfer learning, testing whether pre-training on related supervised tasks could improve performance on BoolQ. They experimented with transferring from extractive QA (SQuAD, NQ), paraphrasing (QQP), and multiple-choice tasks (RACE).
The most successful approach was a multi-step transfer: taking a pre-trained BERT model, fine-tuning it on MultiNLI (Natural Language Inference), and then fine-tuning it on the BoolQ dataset.
Results and Discussion
The results demonstrate that natural yes/no questions are indeed challenging. While a majority baseline (always guessing "Yes") achieves 62.3% accuracy, a standard BERT-Large model reaches 76.9%.
The "secret sauce" for BoolQ performance was Natural Language Inference (NLI). As shown in the results, transferring from MultiNLI provided a significant boost of approximately 3.5 points on top of BERT, reaching 80.4% accuracy. This suggests that NLI tasks teach models to recognize contradictions and neutral statements—skills directly applicable to determining if a passage supports or refutes a boolean question. BERT’s unsupervised pre-training and MultiNLI’s supervised entailment training appear to provide complementary linguistic signals.
Despite these gains, a significant gap remains: human performance is estimated at 90%, leaving a nearly 10% lead over the best-performing models.
Limitations and Future Directions
The current task provides models with the specific paragraph containing the answer. A more realistic challenge would require "open-domain" retrieval, where the model must find the relevant passage within a full Wikipedia article or the entire web. Additionally, about 6% of errors in the gold standard involve genuine ambiguity in the question or passage, suggesting a natural ceiling for accuracy. Future work must also address complex negation, as models still struggle to understand when a positive assertion in a text logically excludes the possibility of the question being true.
Summary updated