Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, Jonathan Berant

Abstract

StrategyQA is a novel question-answering benchmark designed to evaluate "implicit" reasoning capabilities in AI models. While traditional multi-hop datasets often provide explicit roadmaps within the question text, StrategyQA requires models to independently infer a reasoning strategy. This involves decomposing high-level queries—such as "Did Aristotle use a laptop?"—into necessary, verifiable sub-questions. Our findings indicate a significant "strategy gap" between human performance and current state-of-the-art models, primarily driven by challenges in information retrieval and logical synthesis.

Introduction: The Strategy Gap

Most existing multi-hop reasoning datasets are "explicit," meaning the questions contain keywords or structural cues that guide the model toward the answer (e.g., "Who is the mother of the person who directed Inception?"). This allows models to succeed by relying on lexical overlap—matching words in the question to words in the text—rather than performing genuine reasoning.

Implicit vs. Explicit Reasoning: StrategyQA focuses on questions where the reasoning steps are not stated in the query.

The authors of StrategyQA argue that real-world reasoning is fundamentally implicit. To determine if Aristotle used a laptop, a model cannot simply search for a sentence containing both terms. Instead, it must recognize that a temporal comparison is required: "When did Aristotle live?" versus "When was the laptop invented?". StrategyQA addresses this "strategy gap" by forcing models to bridge the distance between a high-level query and the atomic sub-questions needed to solve it.

Methodology

To create a dataset that avoids the pitfalls of word-matching, the authors developed a multi-stage data collection pipeline:

  • Term-based Priming (CQW): Crowdsourcers were given a single Wikipedia term (e.g., "Silk") to spark creative, implicit questions rather than being asked to summarize a specific paragraph.
  • Adversarial Filtering: A "model-in-the-loop" approach was used to ensure difficulty. Questions were only included if they successfully fooled a suite of pre-trained RoBERTa-based solvers.
  • Decomposition and Evidence Annotation (SQD & EVM): For each question, annotators provided a sequence of sub-questions (decomposition) and linked each step to specific evidence paragraphs in Wikipedia.
  • Boolean Constraint: The dataset focuses on Yes/No questions to isolate the challenge of reasoning complexity from the difficulties of open-ended text generation.
The StrategyQA data collection pipeline involving term-based priming, question decomposition, and evidence mapping.

Dataset Characteristics

The primary StrategyQA dataset consists of 2,780 examples, each paired with its strategy decomposition and supporting evidence. To establish baselines, the authors utilized auxiliary training sets including BoolQ (boolean QA), MNLI (natural language inference), and DROP (numerical reasoning). Comparative analysis shows that StrategyQA possesses significantly lower lexical overlap between questions and evidence than datasets like HotpotQA, making traditional information retrieval (IR) methods like BM25 far less effective.

Key Insights and Results

The core finding of the study is that reasoning is not just about connecting facts, but about knowing which facts to look for in the first place.

  • Human vs. Model Performance: Humans achieved 87% accuracy, often leveraging internal background knowledge to skip intermediate steps. In contrast, the best baseline model (RoBERTa fine-tuned on multiple tasks) reached only ~66% accuracy.
  • The Retrieval Bottleneck: Retrieval is a major hurdle for implicit reasoning. When using the original question as a search query, systems found the correct evidence paragraphs only 17.4% of the time (Recall@10).
  • Oracle Gains: Even when models were provided with the "gold" evidence paragraphs, performance only improved to 70.7%. This suggests that even with the correct facts, the logical "hop" required for implicit reasoning remains a significant challenge for current AI systems.

Conclusion

StrategyQA highlights a critical weakness in current QA systems: the inability to autonomously formulate search and reasoning strategies. The results suggest that the next frontier in the field is improving neural retrieval—developing models that can "think" about the information they need before they begin searching. While the dataset is relatively small, it serves as a rigorous benchmark for the "strategy gap" that must be closed to achieve human-level reasoning.

Summary updated

Abstract

A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of ∼66%.
BibTeX
@article{geva2021did,
  author = {Mor Geva and Daniel Khashabi and Elad Segal and Tushar Khot and Dan Roth and Jonathan Berant},
  title = {{Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00370},
}
Figure