Vocabulary Matters: A Simple yet Effective Approach to Paragraph-level Question Generation

Vishwajeet Kumar, Manish Joshi, Ganesh Ramakrishnan, Yuan-Fang Li

Abstract

Most state-of-the-art Question Generation (QG) systems operate at the sentence level, often failing to capture the broader context required for complex information retrieval. This paper presents a "simple yet effective" approach to paragraph-level QG. By augmenting a standard sequence-to-sequence model with a dynamic, paragraph-specific dictionary and a persistent copy attention mechanism, the proposed model, NQG$_{dd}$, achieves a significant 38% improvement in BLEU-4 scores on the SQuAD dataset, outperforming more complex architectures while remaining computationally efficient.

Introduction

Question Generation (QG) is the task of automatically creating questions from a given text snippet. While sentence-level QG has seen success in generating questions for simple facts, real-world applications often require synthesizing information spread across an entire paragraph. Approximately 20% of questions in the Stanford Question Answering Dataset (SQuAD) require this broader context, which sentence-only models are fundamentally unable to provide.

Generating questions from paragraphs is significantly more challenging due to the vastly increased search space. Previous attempts to address this utilized complex neural architectures, such as gated self-attention, but frequently struggled with "out-of-vocabulary" (OOV) words. When a model encounters a rare term or proper noun not seen during training, it typically defaults to a generic <unk> (unknown) token, resulting in vague or nonsensical output. This paper argues that the bottleneck in QG performance is not the complexity of the encoder, but rather how the model manages its vocabulary.

The NQG$_{dd}$ Architecture

The authors propose NQG$_{dd}$, a model built upon a standard Bidirectional Long Short-Term Memory (Bi-LSTM) encoder-decoder architecture. The innovation lies in two primary components designed to handle data sparsity and OOV words:

  1. Dynamic, Paragraph-Specific Dictionary: Instead of relying solely on a static global vocabulary, the model generates a unique "mini-dictionary" for every input paragraph. This dictionary includes every word present in the specific source text, ensuring that technical terms or proper nouns are always "known" locally.
  2. Persistent Copy Attention: The model employs a probabilistic "switch" during the generation phase. At each step, it calculates the probability of either generating a common word from the global vocabulary (e.g., "What," "is") or copying a specific word directly from the input paragraph’s dynamic dictionary.
Figure 1: The NQG_dd architecture featuring the Bi-LSTM encoder-decoder and the copy mechanism for final distribution.

By integrating these components, the model avoids the <unk> token trap and can precisely lift relevant entities from the source text into the generated question. Furthermore, because the architecture avoids heavy self-attention mechanisms, it trains at least twice as fast as previous state-of-the-art models while utilizing fewer parameters.

Experimental Setup

The model was evaluated using the SQuAD dataset, reformatted to provide the model with the entire paragraph as input rather than just the sentence containing the answer. The dataset was split into 77,526 training instances, 9,995 for validation, and 10,556 for the final test set. The model is "answer-aware," meaning the target answer is provided as an input feature to guide the question generation process.

Results and Discussion

The quantitative results demonstrate that the dynamic dictionary and copy mechanism provide a substantial performance boost across all standard NLP metrics.

Figure 4: Quantitative performance comparison of NQG_dd against previous state-of-the-art models.

As shown in the results table, NQG$_{dd}$ achieved a BLEU-4 score of 22.62, representing a 6.24-point (38%) absolute improvement over the previous state-of-the-art (MPGSN). Human evaluation further corroborated these findings. In blind tests, the model with copy attention scored 71 in relevance, compared to just 43 for the version without it. This confirms that the ability to copy specific terms from the source text is crucial for generating questions that are contextually grounded.

Conclusion and Future Work

This research highlights that for certain NLP tasks, targeted architectural tweaks to handle vocabulary sparsity are more effective than simply increasing model depth. While NQG$_{dd}$ represents a major step forward for RNN-based QG, several avenues for future work remain. The study does not explore how these dynamic dictionary concepts translate to Transformer-based architectures like T5 or GPT. Additionally, future iterations could focus on "answer-agnostic" generation, where the model must first identify "question-worthy" segments of a paragraph autonomously.

Summary updated

BibTeX
@inproceedings{kumar2020vocabulary,
  author = {Vishwajeet Kumar and Manish Joshi and Ganesh Ramakrishnan and Yuan-Fang Li},
  title = {{Vocabulary Matters: A Simple yet Effective Approach to Paragraph-level Question Generation}},
  booktitle = {AACL/IJCNLP},
  year = {2020},
  doi = {10.18653/v1/2020.aacl-main.78},
}
Figure