MultiSpanQA: A Dataset for Multi-Span Question Answering

Haonan Li, Martin Tomko, Maria Vasardani, Timothy BaldwinNAACL 2022 · OpenReview

Abstract

Haonan Li, Martin Tomko, Maria Vasardani, Timothy Baldwin. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2022.

Introduction

Most current Reading Comprehension (RC) research operates under a simplifying assumption: the answer to a question exists as a single, contiguous "span" of text within a document. While this works for simple facts, real-world queries often require gathering multiple pieces of information scattered throughout a passage. For example, a question like "Which countries does the Danube River flow through?" results in a list of ten distinct countries separated by various tokens.

Existing datasets often ignore these cases, treat the entire list as one giant span (including irrelevant "noise"), or lack the structural annotations needed for effective training. MultiSpanQA addresses this gap by providing a high-quality, manually re-annotated dataset where each individual component of a multi-part answer is identified.

Methods

The authors treat multi-span QA as a sequence tagging problem, enhanced with global constraints to improve accuracy. The architecture consists of several key components:

  1. Contextualized Encoder: A BERT-based encoder generates representations for the question and passage.
  2. Sequence Tagger: A standard IOB (Inside, Outside, Beginning) tagger identifies potential answer tokens.
  3. Global Predictors: Two additional heads predict the Span Number (total distinct answer spans) and the Answer Structure (logical relationship between spans, such as conjunction or disjunction).
  4. Span Adjustment Module: This module acts as a final filter, ranking identified spans by confidence and keeping only the top-$n$ results based on the predicted span number.
The MultiSpanQA model architecture featuring global predictors and a span adjustment module.

Datasets

The researchers introduced two versions of the MultiSpanQA dataset, derived from Google's Natural Questions (NQ):

  • Basic Version: 6,536 examples where every answer is guaranteed to be multi-span.
  • Expanded Version: Over 19,000 examples, including single-span answers and unanswerable questions, reflecting a more realistic "in-the-wild" scenario.

Answers are categorized by semantics (e.g., Human, Location, Numeric) and logical structure (e.g., Conjunctions or Disjunctions).

Key Results

The proposed "Multi (full)" model significantly outperforms traditional single-span baselines. On the MultiSpanQA basic test set, it achieved an Exact-match F1 of 59.28% and a Partial-match F1 of 76.26%.

Performance comparison of various models on the MultiSpanQA and MultiSpanQA (expand) datasets.

In the expanded version, the model maintained a strong lead, demonstrating that the multi-span architecture remains robust even when faced with simpler, traditional QA tasks.

Discussion and Future Work

A key insight from the study is the "List Advantage": the model performs better on questions with many spans (>7) than on those with only 2 or 3, likely due to predictable syntactic patterns in long lists. However, a primary limitation remains Answer Structure Prediction. While the model excels at finding spans, it struggles to accurately classify the logical relationship between them (achieving 84.38% accuracy, only slightly above a baseline that assumes all answers are lists). Future work will focus on better modeling semantic dependencies between spans to understand how they relate logically.

Summary updated

BibTeX
@inproceedings{li2022multispanqa,
  author = {Haonan Li and Martin Tomko and Maria Vasardani and Timothy Baldwin},
  title = {{MultiSpanQA: A Dataset for Multi-Span Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=reuBhCV92t},
}
Figure