LIQUID: A Framework for List Question Answering Dataset Generation

Seongyun Lee, Hyunjae Kim, Jaewoo Kang

Abstract

List question answering (QA) requires identifying multiple non-contiguous spans of text as answers to a single query. Generating high-quality training data for this task is challenging due to the high cost of human annotation and the difficulty of creating coherent synthetic list questions. We present LIQUID, an automated framework that leverages text summarization, iterative filtering, and answer expansion to generate high-quality list QA datasets from unlabeled corpora. Our approach uses summaries as semantic anchors to identify related entities, ensuring the generation of natural and complex questions. Evaluated across general and biomedical domains, LIQUID-trained models demonstrate significant performance gains, including a 5.0 F1 point improvement on MultiSpanQA and a 2.8 F1 point average increase on BioASQ benchmarks.

Introduction

Extractive Question Answering (QA) has seen remarkable progress, primarily focusing on single-span answers. However, real-world information needs often involve "list" questions—queries that require identifying multiple, non-contiguous spans within a document (e.g., "Which universities were inspired by the Oxford residential system?"). The primary bottleneck for advancing list QA is the scarcity of high-quality training data. Human annotation for list QA is exceptionally labor-intensive, as annotators must exhaustively identify every correct answer span to ensure dataset utility. Existing synthetic data generation methods for single-span QA do not easily translate to list QA; simply selecting random entities often results in trivial or nonsensical questions.

The LIQUID Framework

To address these challenges, we propose LIQUID, a four-stage pipeline designed to transform raw text into high-quality list QA pairs:

  1. Summarization-Based Answer Extraction: LIQUID first summarizes a passage using a BART model. Named entities are then extracted from this summary. The core insight is that entities co-occurring in a summary are inherently semantically related, making them ideal candidates for a cohesive list question.
  2. Question Generation (QG): These candidate entities, along with the original passage, are provided to a T5 model. By concatenating multiple answers in the prompt, the model—even when pre-trained on single-span data—generates high-quality list-style questions.
  3. Iterative Filtering: A separate QA model validates the generated question. If the model lacks confidence in a specific answer from the list, that answer is removed, and the question is re-generated. This loop continues until a stable answer set is achieved.
  4. Answer Expansion: To ensure completeness, the QA model re-scans the original passage for additional spans that answer the generated question but were missed during the initial entity recognition phase.
The LIQUID four-stage pipeline for automated list QA data generation.

Experimental Setup

We evaluated LIQUID using 140,000 synthetic pairs generated from Wikipedia and PubMed corpora. Performance was tested across two distinct domains:

  • General Domain: MultiSpanQA (Wikipedia-based) and Quoref (focused on coreference reasoning).
  • Biomedical Domain: BioASQ benchmarks (7b, 8b, and 9b), which involve complex medical literature.

Results and Analysis

Models trained with LIQUID data consistently outperformed baselines trained solely on human-labeled data. On MultiSpanQA, we observed a 5.0 F1 point improvement. In the biomedical domain, BioASQ scores improved by an average of 2.8 points. Notably, on MultiSpanQA, LIQUID surpassed the previous state-of-the-art multi-task model by 12.1 F1 points using the same base architecture.

F1 Score vs. Data Size for MultiSpanQA and BioASQ 9b, comparing LIQUID's refined data against initial synthetic data.

Key insights include the effectiveness of using summaries as semantic anchors, which prevents the grouping of unrelated entities. Furthermore, the iterative filtering and expansion steps create a "committee" effect where different models verify and complete each other's work, significantly reducing noise in the synthetic dataset.

Discussion and Limitations

While LIQUID represents a significant step forward, it has limitations. The framework's reliance on Named Entity Recognition (NER) makes it less effective for questions requiring descriptive, non-entity answers (e.g., "reasons for a market crash"). Additionally, synthetic data tends to favor shorter lists (2-3 items) compared to the longer lists often found in human-annotated data. Future work will focus on training QG models specifically for list data and improving the expansion phase to handle non-entity answer spans.

Summary updated

Abstract

Question answering (QA) models often rely on large-scale training datasets, which necessitates the development of a data generation framework to reduce the cost of manual annotations. Although several recent studies have aimed to generate synthetic questions with single-span answers, no study has been conducted on the creation of list questions with multiple, non-contiguous spans as answers. To address this gap, we propose LIQUID, an automated framework for generating list QA datasets from unlabeled corpora. We first convert a passage from Wikipedia or PubMed into a summary and extract named entities from the summarized text as candidate answers. This allows us to select answers that are semantically correlated in context and is, therefore, suitable for constructing list questions. We then create questions using an off-the-shelf question generator with the extracted entities and original passage. Finally, iterative filtering and answer expansion are performed to ensure the accuracy and completeness of the answers. Using our synthetic data, we significantly improve the performance of the previous best list QA models by exact-match F1 scores of 5.0 on MultiSpanQA, 1.9 on Quoref, and 2.8 averaged across three BioASQ benchmarks.
BibTeX
@inproceedings{lee2023liquid,
  author = {Seongyun Lee and Hyunjae Kim and Jaewoo Kang},
  title = {{LIQUID: A Framework for List Question Answering Dataset Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26529},
}
Figure