Neural Question Generation with Answer Pivot

Bingning Wang, Xiaochuan Wang, Ting Tao, Qi Zhang, Jingfang XuAAAI 2020 · OpenReview

Abstract

Neural question generation (NQG) aims to produce questions from a given context using deep neural networks. Traditional methods often struggle with a trade-off: answer-aware models produce relevant but trivial questions, while answer-agnostic models generate diverse but often unanswerable ones. We propose a novel "Answer Pivot" model that treats the answer as a hidden intermediate step. By predicting a candidate answer span internally, our joint model bridges the gap between these two approaches, achieving state-of-the-art results on the SQuAD dataset and demonstrating significant utility for downstream machine reading comprehension tasks.

Introduction

In Machine Reading Comprehension (MRC), generating high-quality questions is a challenging task. Existing NQG methods generally fall into two categories:

  1. Answer-Aware NQG: These models are provided with the target answer. While they produce answerable questions, they are often simple syntactic transformations of the source sentence, heavily biased toward named entities.
  2. Answer-Agnostic NQG: These models generate questions without prior knowledge of the answer. While they produce more diverse queries, they frequently fail to generate questions that can actually be answered by the text.

Our research addresses this "fluency vs. relevance" trade-off by introducing a joint model that predicts a candidate answer span—the "pivot"—before generating the question.

Methodology

We introduce a Transformer-based architecture centered around a Pivot Answer Predictor.

The Pivot Mechanism and STE

Instead of requiring an answer as input, the model predicts a binary label for every word in the paragraph to identify potential answer spans. Because this involves a discrete binary choice (0 or 1), standard backpropagation is not feasible. We employ a Straight-Through Estimator (STE) to approximate gradients, allowing for end-to-end training. To prevent "posterior collapse"—where the generator ignores the pivot—we incorporate an auxiliary loss that ensures the generated question remains grounded in the predicted answer span.

Key Results

Our "Answer Pivot" model was evaluated on the SQuAD dataset. As shown in the table below, our model consistently outperforms previous benchmarks across all standard automatic metrics, including BLEU, Meteor, and Rouge-L, in both answer-agnostic and answer-aware settings.

Table 1: Performance comparison of our model against various benchmarks on the SQuAD dataset.

Human evaluation further confirmed these findings. Volunteers rated our model's questions significantly higher for Significance (focusing on important text segments) compared to answer-aware models, and higher for Answerability compared to answer-agnostic baselines.

Impact on Downstream Tasks

A key advantage of high-quality NQG is the ability to generate synthetic training data for other MRC models. We used our model to generate 80,000 additional question-answer pairs to augment the training of a BERT-base model.

Figure 1: Improvement in BERT-base performance (EM and F1 scores) on SQuAD as more synthetic data from our model is added.

As illustrated in Figure 1, the inclusion of our generated data led to significant improvements in both Exact Match (EM) and F1 scores, outperforming data generated by other models like NQG++ and SynNet. This demonstrates that our model produces questions that are not only fluent but also semantically rich enough to serve as effective training signals.

Conclusion

By treating the answer as a hidden pivot, our model successfully combines the strengths of answer-aware and answer-agnostic NQG. The pivot predictor even demonstrates an ability to perform unsupervised answer selection, focusing on significant phrases without explicit labels. While complex multi-sentence reasoning remains a challenge for future work, the Answer Pivot approach represents a significant step forward in generating meaningful, answerable questions for machine reading comprehension.

Summary updated

Abstract

Neural question generation (NQG) is the task of generating questions from the given context with deep neural networks. Previous answer-aware NQG methods suffer from the problem that the generated answers are focusing on entity and most of the questions are trivial to be answered. The answer-agnostic NQG methods reduce the bias towards named entities and increasing the model's degrees of freedom, but sometimes result in generating unanswerable questions which are not valuable for the subsequent machine reading comprehension system. In this paper, we treat the answers as the hidden pivot for question generation and combine the question generation and answer selection process in a joint model. We achieve the state-of-the-art result on the SQuAD dataset according to automatic metric and human evaluation.
BibTeX
@inproceedings{wang2020neural,
  author = {Bingning Wang and Xiaochuan Wang and Ting Tao and Qi Zhang and Jingfang Xu},
  title = {{Neural Question Generation with Answer Pivot}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=FEXbL9sb3b},
}
Figure