BiQuAD: Towards QA based on deeper text understanding

Frank Grimm, Philipp CimianoACL 2021

TL;DR

This paper introduces BiQuAD, a new question-answering dataset designed to push beyond simple text span extraction towards deeper text comprehension and reasoning. Current state-of-the-art models struggle significantly with BiQuAD, achieving drastically lower scores than on existing benchmarks, highlighting a critical gap in their ability to truly understand and infer information.

What Problem Does This Paper Address?

Modern question-answering (QA) systems, particularly those based on large transformer models, have achieved remarkable success on benchmarks like SQuAD. However, this success often stems from their ability to identify answer spans within a text rather than truly understanding the underlying meaning. The authors argue that existing benchmarks may not adequately test deeper comprehension, leading to models that exploit surface-level cues and artifacts rather than genuine reasoning. This paper aims to address this by creating a dataset that necessitates more complex inference, aggregation, and deductive reasoning, moving beyond simple extractive QA.

Methods

The core of this work is the creation of the BiQuAD dataset. The authors leverage a structured database of soccer matches (ESDB) containing minute-by-minute event data. This data is transformed into two parallel representations:

  1. Datalog Programs: These represent a logical model of the match events, enabling structured querying and aggregation.
  2. Natural Language Match Reports: These are synthetically generated using a set of handcrafted templates, creating long-form, factual descriptions of the matches.

Questions are generated using templates that are designed to probe different levels of understanding, including simple facts, multi-entity relations, temporal reasoning, and aggregation. Crucially, a subset of questions are deliberately made unanswerable to test robustness. The authors also provide baseline results using state-of-the-art transformer models like ALBERT, fine-tuned on their dataset.

Datasets

  • BiQuAD: The primary contribution is this newly introduced dataset. It consists of 4,190 synthetic soccer match reports and 99,149 question-answer pairs. The dataset is designed to require deeper comprehension beyond span extraction, featuring both answerable and unanswerable questions.
  • SQuAD 2.0: Used for transfer learning experiments, where models pre-trained on SQuAD 2.0 are evaluated on BiQuAD to assess their generalization capabilities.

Key Insights and Ideas

  • The "Extractive Trap": Current QA models excel at finding answer spans but struggle with tasks requiring inference, aggregation, or temporal reasoning. BiQuAD's design explicitly targets these weaknesses.
  • Synthetic Data for Targeted Evaluation: By generating data from a structured knowledge base (Datalog) and then rendering it into natural language, the authors can precisely control the complexity and type of reasoning required, enabling detailed error analysis.
  • Bridging Structured and Unstructured Data: The parallel Datalog and natural language representations offer a unique opportunity to study how models align their internal representations with explicit knowledge structures, aiding model introspection.
  • Unanswerable Questions as a Robustness Test: The inclusion of adversarial unanswerable questions, sampled from other documents, helps ensure models aren't simply exploiting superficial cues that might coincidentally match.

Key Results

  • State-of-the-art transformer models (like ALBERT) achieve surprisingly low F₁ scores on BiQuAD, around 39.0, a stark contrast to their performance on SQuAD 2.0 (around 93.0).
  • Even when models are fine-tuned on BiQuAD, performance remains low across many question template categories, particularly those involving temporal reasoning.
  • Transfer learning from SQuAD 2.0 shows that models can determine answerability reasonably well, suggesting they can identify some superficial cues, but this ability breaks down for more complex reasoning categories.

Limitations and Future Work

  • Synthetic Text Style: The authors acknowledge that the synthetic nature of the text, generated by templates, results in an unnaturally factual and sober style. This might be exploited by heuristics or models specifically tuned to this style.
  • Domain Specificity: While the soccer domain is rich in events and relations, it is still a specific domain. The authors suggest extending the approach to other domains to improve generalizability.
  • Future Work: The authors plan to further develop tooling for model introspection and explore additional domains. They aim to establish synthetic corpora like BiQuAD as valuable tools for understanding and improving reasoning in QA systems, complementing natural text datasets.

Summary updated

BibTeX
@inproceedings{grimm2021biquad,
  author = {Frank Grimm and Philipp Cimiano},
  title = {{BiQuAD: Towards QA based on deeper text understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure