Abstract
List question answering (QA) requires identifying multiple non-contiguous spans of text as answers to a single query. Generating high-quality training data for this task is challenging due to the high cost of human annotation and the difficulty of creating coherent synthetic list questions. We present LIQUID, an automated framework that leverages text summarization, iterative filtering, and answer expansion to generate high-quality list QA datasets from unlabeled corpora. Our approach uses summaries as semantic anchors to identify related entities, ensuring the generation of natural and complex questions. Evaluated across general and biomedical domains, LIQUID-trained models demonstrate significant performance gains, including a 5.0 F1 point improvement on MultiSpanQA and a 2.8 F1 point average increase on BioASQ benchmarks.
Introduction
Extractive Question Answering (QA) has seen remarkable progress, primarily focusing on single-span answers. However, real-world information needs often involve "list" questions—queries that require identifying multiple, non-contiguous spans within a document (e.g., "Which universities were inspired by the Oxford residential system?"). The primary bottleneck for advancing list QA is the scarcity of high-quality training data. Human annotation for list QA is exceptionally labor-intensive, as annotators must exhaustively identify every correct answer span to ensure dataset utility. Existing synthetic data generation methods for single-span QA do not easily translate to list QA; simply selecting random entities often results in trivial or nonsensical questions.
The LIQUID Framework
To address these challenges, we propose LIQUID, a four-stage pipeline designed to transform raw text into high-quality list QA pairs:
- Summarization-Based Answer Extraction: LIQUID first summarizes a passage using a BART model. Named entities are then extracted from this summary. The core insight is that entities co-occurring in a summary are inherently semantically related, making them ideal candidates for a cohesive list question.
- Question Generation (QG): These candidate entities, along with the original passage, are provided to a T5 model. By concatenating multiple answers in the prompt, the model—even when pre-trained on single-span data—generates high-quality list-style questions.
- Iterative Filtering: A separate QA model validates the generated question. If the model lacks confidence in a specific answer from the list, that answer is removed, and the question is re-generated. This loop continues until a stable answer set is achieved.
- Answer Expansion: To ensure completeness, the QA model re-scans the original passage for additional spans that answer the generated question but were missed during the initial entity recognition phase.
Experimental Setup
We evaluated LIQUID using 140,000 synthetic pairs generated from Wikipedia and PubMed corpora. Performance was tested across two distinct domains:
- General Domain: MultiSpanQA (Wikipedia-based) and Quoref (focused on coreference reasoning).
- Biomedical Domain: BioASQ benchmarks (7b, 8b, and 9b), which involve complex medical literature.
Results and Analysis
Models trained with LIQUID data consistently outperformed baselines trained solely on human-labeled data. On MultiSpanQA, we observed a 5.0 F1 point improvement. In the biomedical domain, BioASQ scores improved by an average of 2.8 points. Notably, on MultiSpanQA, LIQUID surpassed the previous state-of-the-art multi-task model by 12.1 F1 points using the same base architecture.
Key insights include the effectiveness of using summaries as semantic anchors, which prevents the grouping of unrelated entities. Furthermore, the iterative filtering and expansion steps create a "committee" effect where different models verify and complete each other's work, significantly reducing noise in the synthetic dataset.
Discussion and Limitations
While LIQUID represents a significant step forward, it has limitations. The framework's reliance on Named Entity Recognition (NER) makes it less effective for questions requiring descriptive, non-entity answers (e.g., "reasons for a market crash"). Additionally, synthetic data tends to favor shorter lists (2-3 items) compared to the longer lists often found in human-annotated data. Future work will focus on training QG models specifically for list data and improving the expansion phase to handle non-entity answer spans.
Summary updated