IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

Wenhao Yu, Meng Jiang, Peter Clark, Ashish Sabharwal

Abstract

Open-domain question answering (ODQA) systems have achieved remarkable success in retrieving and processing factual information. However, these systems often falter when faced with counterfactual reasoning—the ability to imagine alternatives to reality and deduce their consequences. This paper introduces IfQA, a new benchmark designed to evaluate ODQA systems under counterfactual presuppositions. Comprising over 3,800 questions, IfQA requires models to navigate a complex "retrieve-then-reason" pipeline. Our findings indicate that even state-of-the-art large language models (LLMs) struggle with this task, as it necessitates a delicate balance between factual retrieval and hypothetical intervention.

Introduction: The "What If" Challenge

In the current landscape of artificial intelligence, models excel at answering factual queries such as "What is the capital of France?" by searching vast repositories of information. However, when presented with a counterfactual scenario—for example, "If Los Angeles were on the East Coast, what would be the time difference between LA and Paris?"—traditional ODQA frameworks often fail. This task requires counterfactual reasoning, a cognitive process that involves imagining a change to a known fact and determining the downstream effects of that change.

IfQA is the first large-scale open-domain dataset specifically focused on these counterfactual presuppositions. It moves beyond simple fact retrieval, challenging models to reason through imagined, non-factual interventions while grounded in a factual world.

The Problem: Beyond Simple Retrieval

Most modern ODQA systems function as sophisticated search engines. They are optimized to find a specific fact that exists verbatim on the web. Counterfactual questions, however, involve a causal intervention that renders direct retrieval insufficient. To answer a counterfactual question correctly, a system must perform a multi-step process:

  1. Identify Factual Background: Determine the current reality (e.g., Mount Everest's actual height).
  2. Apply Intervention: Modify that reality based on the question's premise (e.g., "If Everest were 300 meters shorter...").
  3. Comparative Reasoning: Evaluate the new state against other global facts (e.g., "Is it still the tallest mountain in the world?").

Existing benchmarks do not adequately test this pipeline, leaving a gap in our ability to measure true machine intelligence and logical flexibility.

Methodology

The researchers developed a rigorous three-phase pipeline to construct the IfQA dataset:

  • Passage Selection: Using heuristic keyword searches (e.g., "originally," "due to," "caused by"), the team identified Wikipedia passages describing causal events, which serve as the foundation for counterfactual scenarios.
  • Crowdsourced Annotation: Workers wrote questions based on these passages. To ensure linguistic diversity and prevent repetitive patterns, the researchers rotated example prompts during the annotation process.
  • Verification: A separate group of annotators validated the questions for readability, the necessity of the source passage, and the accuracy of the provided answers.

Dataset Overview

The resulting IfQA dataset contains over 3,800 questions annotated against Wikipedia passages. The dataset is divided into two primary splits:

  • IfQA-S (Supervised): Intended for training and testing models in a standard supervised learning environment.
  • IfQA-F (Few-shot): Designed for evaluating LLMs with minimal prompting examples.

The questions are categorized by answer type, with Entities making up 49.7%, followed by Numbers (15.9%), Dates (14.5%), and other miscellaneous types (19.9%).

Key Insights and Experimental Results

Evaluation of both closed-book models (relying on internal parameters) and open-book models (using retrieval) revealed several critical challenges:

The Retrieval Paradox

Surprisingly, traditional sparse retrieval methods like BM25 outperformed modern dense retrievers (DPR). Dense retrievers typically rely on semantic similarity; however, in counterfactual scenarios, the question often contains terms that explicitly contradict the factual evidence in the gold passage. This contradiction confuses vector-based matching, whereas keyword-based retrieval remains more robust.

The Hallucination Trap and Reasoning Bottlenecks

Closed-book models, such as GPT-3, frequently suffer from hallucinations when faced with counterfactuals. Instead of reasoning from the provided text, they often invent entirely new historical narratives to satisfy the prompt's premise. Furthermore, even when the correct passage is provided to the model, performance remains relatively low. This suggests that the primary bottleneck is not just information retrieval, but the logical "math" required to process the counterfactual intervention.

Performance Metrics

The benchmark proved highly difficult for current architectures:

  • Supervised Pipeline: The best supervised model (FiD-l) achieved an F1 score of 44.5.
  • Few-shot LLMs: GPT-3 using Chain-of-Thought (CoT) prompting reached only 27.4 Exact Match (EM).
  • Hybrid Success: The highest performance (55.27 F1) was achieved by a hybrid approach: using BM25 to retrieve relevant passages and then feeding those passages to GPT-3 for reasoning. This highlights the necessity of grounding LLMs in factual text to mitigate hallucinations during counterfactual tasks.

Limitations and Future Work

While IfQA provides a robust foundation for counterfactual ODQA, it has limitations. The reliance on Wikipedia results in a focus on historical events, sports, and geography, with less emphasis on abstract or scientific counterfactuals. Additionally, heuristic filtering may miss subtle causal relationships.

Future research should focus on developing "counterfactual-aware" retrievers capable of handling the semantic discrepancies between a hypothetical question and factual evidence. Improving the reasoning capabilities of LLMs to handle comparative logic after an intervention remains a vital frontier for the field.

Summary updated

Abstract

Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability. To address this void, we introduce the first such dataset, named IfQA, where each question is based on a counterfactual presupposition via an "if" clause. For example, if Los Angeles was on the east coast of the U.S., what would be the time difference between Los Angeles and Paris? Such questions require models to go beyond retrieving direct factual knowledge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters. The IfQA dataset contains over 3,800 questions that were annotated annotated by crowdworkers on relevant Wikipedia passages. Empirical analysis reveals that the IfQA dataset is highly challenging for existing open-domain QA methods, including supervised retrieve-then-read pipeline methods (EM score 36.2), as well as recent few-shot approaches such as chain-of-thought prompting with GPT-3 (EM score 27.4). The unique challenges posed by the IfQA benchmark will push open-domain QA research on both retrieval and counterfactual reasoning fronts.
BibTeX
@inproceedings{yu2023ifqa,
  author = {Wenhao Yu and Meng Jiang and Peter Clark and Ashish Sabharwal},
  title = {{IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.515},
}
Figure