Generative Language Models for Paragraph-Level Question Generation

Asahi Ushio, Fernando Alva-Manchego, José Camacho-Collados

Abstract

Question Generation (QG) is a critical task for educational technology and data augmentation, yet the field has historically lacked standardized evaluation protocols and benchmarks. This paper presents QG-Bench, a unified multilingual and multi-domain framework for QG. Through extensive experiments with generative models like T5 and BART, the authors demonstrate that paragraph-level context and systematic hyperparameter optimization lead to state-of-the-art performance. The study also highlights the limitations of traditional overlap-based metrics, advocating for semantic-aware evaluation in QG research.

Introduction

Question Generation (QG) involves automatically creating a question based on a text passage and a specific answer. This capability is vital for building educational tools, improving search engines, and augmenting data for training Question Answering (QA) systems.

Despite its importance, the field has been fragmented by the use of disparate datasets, varying input formats (sentence-level vs. paragraph-level), and inconsistent evaluation protocols. Furthermore, standard automated metrics like BLEU or ROUGE often correlate poorly with human judgment in creative tasks. This paper seeks to standardize the field by providing a unified benchmark and a robust analysis of modern generative language models across multiple languages and domains.

Overview of the Question Generation task using paragraph-level context and answer highlighting.

Methodology and Model Optimization

The authors treat QG as a sequence-to-sequence (seq2seq) task, utilizing T5 and BART for English, and mT5 and mBART for multilingual applications. To provide the model with specific answer context, they employ a "highlight token" (<hl>) to demarcate the answer within the paragraph: [context_before] <hl> [answer] <hl> [context_after].

A key contribution of this work is a rigorous "two-phase" hyperparameter search. By testing 24 different configurations for each model to optimize learning rates, batch sizes, and label smoothing, the authors found that simple optimization significantly boosted performance. This approach allowed even smaller models to outperform previous specialized state-of-the-art architectures.

QG-Bench: A Unified Benchmark

The authors compiled QG-Bench by unifying several existing QA datasets into a standardized QG format:

  1. SQuAD (English): The standard Wikipedia-based benchmark.
  2. SQuADShifts: Used to test domain adaptation across Amazon reviews, NYT articles, and Reddit posts.
  3. SubjQA: Designed to test "subjective" QG, such as questions about opinions in reviews.
  4. Multilingual QG: Datasets spanning eight languages, including Japanese, Spanish, German, Russian, Korean, Italian, and French.

Performance and Key Insights

The Importance of Context and Tuning

Models perform significantly better when provided with full paragraph context rather than just the single sentence containing the answer. This "global" context helps the model resolve ambiguities and generate more coherent questions. Furthermore, the results indicate that rigorous hyperparameter tuning is often more impactful than raw model size. For instance, T5-Small (60M parameters) performed competitively with much larger previous models like UniLMv2 (110M parameters).

Performance comparison of optimized T5 and BART models against existing state-of-the-art QG architectures.

The "BLEU" Trap

The study confirms that BLEU4 and ROUGE-L—the most commonly used metrics in QG—are unreliable indicators of quality. Instead, METEOR and MoverScore correlate much more closely with human judgments of "answerability" (whether the question can actually be answered by the text). In manual evaluations, T5-Large emerged as the overall winner, achieving near-perfect scores for grammaticality (2.93/3.0) and high marks for answerability (2.80/3.0).

Domain Adaptation and Multilingual Challenges

For domain adaptation (e.g., moving from Wikipedia to Reddit), the most effective strategy is initializing a model on a large general dataset like SQuAD before fine-tuning on the target domain. While models performed well in English, Spanish, and Russian, they struggled in languages like German and French, likely due to smaller available training sets.

Conclusion and Future Directions

While the optimized generative models set new benchmarks, limitations remain. Current models are restricted to roughly 500 tokens, preventing QG based on entire documents. Additionally, the benchmark focuses on "extractive" QG where the answer is a literal string; "multi-hop" reasoning remains a challenge. Future work should explore end-to-end systems that incorporate Answer Selection—deciding which parts of a text are worth asking about—and improving robustness for low-resource languages.

Summary updated

Abstract

Powerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we introduce QG-Bench, a multilingual and multidomain benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general-purpose datasets such as SQuAD for English, datasets from ten domains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analysis of the capabilities of language models for the task. First, we propose robust QG baselines based on fine-tuning generative language models. Then, we complement automatic evaluation based on standard metrics with an extensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptability of these models as well as the effectiveness of multilingual models in languages other than English. QG-Bench is released along with the fine-tuned models presented in the paper https://github.com/asahi417/lm-question-generation, which are also available as a demo https://autoqg.net/.
BibTeX
@inproceedings{ushio2022generative,
  author = {Asahi Ushio and Fernando Alva-Manchego and José Camacho-Collados},
  title = {{Generative Language Models for Paragraph-Level Question Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.42},
}
Figure