OneStop QAMaker: Extract Question-Answer Pairs from Text in a One-Stop Approach

Shaobo Cui, Xintong Bao, Xinxing Zu, Yangyang Guo, Zhongzhou Zhao, Ji Zhang, Haiqing ChenarXiv 2021 · OpenReview

Abstract

Large-scale question-answer (QA) pairs are critical for advancing research areas like machine reading comprehension and question answering. To construct QA pairs from documents requires determining how to ask a question and what is the corresponding answer. Existing methods for QA pair generation usually follow a pipeline approach. Namely, they first choose the most likely candidate answer span and then generate the answer-specific question. This pipeline approach, however, is undesired in mining the most appropriate QA pairs from documents since it ignores the connection between question generation and answer extraction, which may lead to incompatible QA pair generation, i.e., the selected answer span is inappropriate for question generation. However, for human annotators, we take the whole QA pair into account and consider the compatibility between question and answer. Inspired by such motivation, instead of the conventional pipeline approach, we propose a model named OneStop to generate QA pairs from documents in a one-stop approach. Specifically, questions and their corresponding answer spans are extracted simultaneously, and the process of question generation and answer extraction mutually affect each other. Additionally, OneStop is much more efficient to be trained and deployed in industrial scenarios since it involves only one model to solve the complex QA generation task. We conduct comprehensive experiments on three large-scale machine reading comprehension datasets: SQuAD, NewsQA, and DuReader. The experimental results demonstrate that our OneStop model outperforms the baselines significantly regarding the quality of generated questions, quality of generated question-answer pairs, and model efficiency.

Introduction

Generating high-quality question-answer (QA) pairs from raw text is a foundational task for training Machine Reading Comprehension (MRC) systems. Traditionally, this has been treated as a multi-stage pipeline problem. Two common workflows exist: D2A2Q (Document → Answer → Question), where a model extracts a candidate answer span before a second model generates a question for it; and D2Q2A (Document → Question → Answer), where a question is generated first and an MRC model subsequently finds the answer.

Comparison of traditional pipeline approaches (D2A2Q, D2Q2A) versus the proposed OneStop approach.

These pipeline approaches suffer from a fundamental flaw: incompatibility. Because the answer extraction and question generation models are trained independently, they lack a shared understanding of what makes a "good" pair. An answer extractor might select a span that is linguistically unsuitable for a coherent question, or a generator might produce a query that cannot be accurately mapped back to the source text. Furthermore, these multi-model systems are computationally expensive and prone to cumulative error.

Methodology

To address these issues, we propose OneStop, a unified framework built on a pre-trained sequence-to-sequence Transformer (such as BART). Unlike pipelines, OneStop performs both tasks in a single forward pass, mimicking the way human annotators consider the compatibility of a question and its answer simultaneously.

Architecture and the "One-Stop" Mechanism

The model utilizes a standard bidirectional encoder to process the document and an auto-regressive decoder to generate the question. The key innovation lies in how the answer is extracted. Instead of a separate model, OneStop uses the decoder’s final hidden state at the <eos> (end-of-sentence) token as a dense representation of the entire generated question. This vector serves as a query for a pointer-style search over the encoder’s hidden states to identify the start and end indices of the answer span.

The OneStop architecture, showing how the decoder's final hidden state informs the Answer Span Predictor.

Joint Optimization

The model is trained using a multi-task loss function that balances language modeling ($\Phi_{lm}$) and answer span extraction ($\Phi_{start}$ and $\Phi_{end}$):

$$\Phi = \lambda \cdot \Phi_{lm} + (1 - \lambda) \cdot (\Phi_{start} + \Phi_{end})$$

This joint training creates a mutual enhancement effect: the answer extraction task forces the generator to produce "answerable" questions, while the generation task guides the extractor toward the most "question-worthy" segments of the document.

Experiments and Results

We evaluated OneStop on three major benchmarks: SQuAD (Wikipedia), NewsQA (CNN articles), and DuReader (Chinese search queries).

Performance and Quality

  • Human Evaluation: In blind tests, OneStop achieved a quality score of 1.41/2.0, significantly higher than the 0.20 scored by traditional pipeline baselines (e.g., DeepNQG + BERT-MRC).
  • Question Quality: OneStop maintained BLEU and ROUGE scores comparable to state-of-the-art standalone question generation models, proving that multi-tasking does not degrade the linguistic quality of the questions.
  • Efficiency: By consolidating two tasks into one model, OneStop significantly reduces the parameter footprint. On SQuAD, OneStop requires approximately 142M parameters, compared to over 248M for a standard BART-QG and BERT-MRC pipeline.

Conclusion

OneStop QAMaker demonstrates that question generation and answer extraction are deeply interconnected tasks that benefit from joint modeling. By moving away from disjointed pipelines, OneStop produces more compatible QA pairs with greater computational efficiency. Future work will focus on incorporating copy mechanisms to handle technical terminology and extending the model to generate multiple diverse QA pairs from longer documents.

Summary updated

Abstract

Large-scale question-answer (QA) pairs are critical for advancing research areas like machine reading comprehension and question answering. To construct QA pairs from documents requires determining how to ask a question and what is the corresponding answer. Existing methods for QA pair generation usually follow a pipeline approach. Namely, they first choose the most likely candidate answer span and then generate the answer-specific question. This pipeline approach, however, is undesired in mining the most appropriate QA pairs from documents since it ignores the connection between question generation and answer extraction, which may lead to incompatible QA pair generation, i.e., the selected answer span is inappropriate for question generation. However, for human annotators, we take the whole QA pair into account and consider the compatibility between question and answer. Inspired by such motivation, instead of the conventional pipeline approach, we propose a model named OneStop generate QA pairs from documents in a one-stop approach. Specifically, questions and their corresponding answer span is extracted simultaneously and the process of question generation and answer extraction mutually affect each other. Additionally, OneStop is much more efficient to be trained and deployed in industrial scenarios since it involves only one model to solve the complex QA generation task. We conduct comprehensive experiments on three large-scale machine reading comprehension datasets: SQuAD, NewsQA, and DuReader. The experimental results demonstrate that our OneStop model outperforms the baselines significantly regarding the quality of generated questions, quality of generated question-answer pairs, and model efficiency.
BibTeX
@misc{cui2021onestop,
  author = {Shaobo Cui and Xintong Bao and Xinxing Zu and Yangyang Guo and Zhongzhou Zhao and Ji Zhang and Haiqing Chen},
  title = {{OneStop QAMaker: Extract Question-Answer Pairs from Text in a One-Stop Approach}},
  year = {2021},
  eprint = {2102.12128},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.12128},
}
Figure