Question Generation for Supporting Informational Query Intents

Xusen Yin, Jonathan May, Li Zhou, Kevin Small

Introduction

Modern Question Answering (QA) systems primarily excel at responding to narrow, factoid-based queries (e.g., "When was a specific person born?"). However, to foster deeper exploration of informational topics, there is a growing need for "suggested questions" (SQs) that guide users toward a more comprehensive understanding of a subject. Existing Question Generation (QG) models, often trained on reading comprehension datasets like SQuAD, typically produce context-dependent questions that are difficult to understand in isolation. This paper proposes a novel approach to generate self-explanatory and introductory questions designed for human interaction rather than just data augmentation.

Methodology: The BERTPGN Architecture

The authors introduce the BERT-based Pointer-Generator Network (BERTPGN), specifically engineered to handle long document contexts. The architecture consists of a pre-trained BERT encoder and a Transformer-based decoder.

The BERTPGN architecture, featuring a BERT encoder and a Transformer-based pointer-generator decoder.

The encoder utilizes type embeddings to distinguish between general context and target answer spans. The decoder employs a pointer-generator mechanism, allowing the model to either generate tokens from a fixed vocabulary or "copy" them directly from the source text. This copy mechanism is vital for handling rare entities or specific terminology found in long documents. To ensure consistency in subword tokenization and improve generalization, the authors use a separate pre-trained BERT model as the language model for the decoder, ensuring the encoder and decoder share the same vocabulary space.

Datasets and Evaluation Strategy

The model is primarily trained on the Natural Questions (NQ) dataset. Unlike SQuAD, NQ questions originate from actual Google search logs, meaning they were asked before the user saw the document. This makes them naturally self-explanatory. The researchers also evaluate the model's out-of-domain performance using a custom News-LA dataset derived from CNN/Daily Mail articles to test if the model can generate broad, article-level questions in a different domain.

A key challenge in QG is evaluating performance on out-of-domain text where reference "gold standard" questions are unavailable. The authors address this by using a pre-trained QA model as an automated evaluator, measuring two primary metrics:

  1. Answerability ($s_{ans}$): The ability of the QA model to find an answer to the generated question.
  2. Granularity ($s_{gra}$): Whether the QA model identifies a long passage (introductory) or a short span (factoid).

Experimental Results

The BERTPGN model demonstrates significant improvements over existing Transformer baselines. On the NQ dataset, it achieves state-of-the-art results for long-answer generation.

Performance comparison of BERTPGN against Transformer baselines across BLEU (B1, B4), METEOR (ME), and ROUGE-L (RL) metrics.

As shown in the results table, the model reached a BLEU-4 score of 20.1 for long-answer (LA) generation, representing a 6.2-point increase over previous benchmarks. For short-answer (SA) generation, it achieved a score of 28.1. Human evaluations further supported these findings, with 62% of NQ-trained questions being rated as context-independent, compared to only 30% for SQuAD-trained baselines. Furthermore, ablation studies highlighted that a full 12-layer BERT encoder is crucial; swapping it for a shallower 2-layer transformer severely degraded performance when encoding document-level context.

Conclusion

By leveraging search-log data and a robust BERT-based architecture, this research successfully generates questions that are both self-explanatory and introductory. While the automated evaluation is limited by the underlying QA model's intelligence, the approach provides a scalable way to enhance informational query intents in search engines and news chatbots. Future work will focus on maintaining neutrality and avoiding information bias in generated suggestions to ensure high journalistic standards.

Summary updated

BibTeX
@inproceedings{yin2021question,
  author = {Xusen Yin and Jonathan May and Li Zhou and Kevin Small},
  title = {{Question Generation for Supporting Informational Query Intents}},
  booktitle = {DialDoc @ ACL},
  year = {2021},
  doi = {10.18653/v1/2021.dialdoc-1.11},
}
Figure