Abstract
Question Generation (QG) is a critical task for educational technology and data augmentation, yet the field has historically lacked standardized evaluation protocols and benchmarks. This paper presents QG-Bench, a unified multilingual and multi-domain framework for QG. Through extensive experiments with generative models like T5 and BART, the authors demonstrate that paragraph-level context and systematic hyperparameter optimization lead to state-of-the-art performance. The study also highlights the limitations of traditional overlap-based metrics, advocating for semantic-aware evaluation in QG research.
Introduction
Question Generation (QG) involves automatically creating a question based on a text passage and a specific answer. This capability is vital for building educational tools, improving search engines, and augmenting data for training Question Answering (QA) systems.
Despite its importance, the field has been fragmented by the use of disparate datasets, varying input formats (sentence-level vs. paragraph-level), and inconsistent evaluation protocols. Furthermore, standard automated metrics like BLEU or ROUGE often correlate poorly with human judgment in creative tasks. This paper seeks to standardize the field by providing a unified benchmark and a robust analysis of modern generative language models across multiple languages and domains.
Methodology and Model Optimization
The authors treat QG as a sequence-to-sequence (seq2seq) task, utilizing T5 and BART for English, and mT5 and mBART for multilingual applications. To provide the model with specific answer context, they employ a "highlight token" (<hl>) to demarcate the answer within the paragraph: [context_before] <hl> [answer] <hl> [context_after].
A key contribution of this work is a rigorous "two-phase" hyperparameter search. By testing 24 different configurations for each model to optimize learning rates, batch sizes, and label smoothing, the authors found that simple optimization significantly boosted performance. This approach allowed even smaller models to outperform previous specialized state-of-the-art architectures.
QG-Bench: A Unified Benchmark
The authors compiled QG-Bench by unifying several existing QA datasets into a standardized QG format:
- SQuAD (English): The standard Wikipedia-based benchmark.
- SQuADShifts: Used to test domain adaptation across Amazon reviews, NYT articles, and Reddit posts.
- SubjQA: Designed to test "subjective" QG, such as questions about opinions in reviews.
- Multilingual QG: Datasets spanning eight languages, including Japanese, Spanish, German, Russian, Korean, Italian, and French.
Performance and Key Insights
The Importance of Context and Tuning
Models perform significantly better when provided with full paragraph context rather than just the single sentence containing the answer. This "global" context helps the model resolve ambiguities and generate more coherent questions. Furthermore, the results indicate that rigorous hyperparameter tuning is often more impactful than raw model size. For instance, T5-Small (60M parameters) performed competitively with much larger previous models like UniLMv2 (110M parameters).
The "BLEU" Trap
The study confirms that BLEU4 and ROUGE-L—the most commonly used metrics in QG—are unreliable indicators of quality. Instead, METEOR and MoverScore correlate much more closely with human judgments of "answerability" (whether the question can actually be answered by the text). In manual evaluations, T5-Large emerged as the overall winner, achieving near-perfect scores for grammaticality (2.93/3.0) and high marks for answerability (2.80/3.0).
Domain Adaptation and Multilingual Challenges
For domain adaptation (e.g., moving from Wikipedia to Reddit), the most effective strategy is initializing a model on a large general dataset like SQuAD before fine-tuning on the target domain. While models performed well in English, Spanish, and Russian, they struggled in languages like German and French, likely due to smaller available training sets.
Conclusion and Future Directions
While the optimized generative models set new benchmarks, limitations remain. Current models are restricted to roughly 500 tokens, preventing QG based on entire documents. Additionally, the benchmark focuses on "extractive" QG where the answer is a literal string; "multi-hop" reasoning remains a challenge. Future work should explore end-to-end systems that incorporate Answer Selection—deciding which parts of a text are worth asking about—and improving robustness for low-resource languages.
Summary updated