Abstract
With the boom of digital educational materials and scalable e-learning systems, the potential for realising AI-assisted personalised learning has skyrocketed. In this landscape, the automatic generation of educational questions will play a key role, enabling scalable self-assessment when a global population is manoeuvring their personalised learning journeys. We develop EduQG, a novel educational question generation model built by adapting a large language model. Our initial experiments demonstrate that EduQG can produce superior educational questions by pre-training on scientific text.
Introduction
As digital e-learning scales, there is a growing need for "intelligent tutors" that can automatically generate high-quality assessment questions. Most current Question Generation (QG) models are trained on general-purpose datasets like SQuAD (Stanford Question Answering Dataset), which is based on Wikipedia articles. While effective for general reading comprehension, these models often lack the depth required for rigorous academic subjects like physics, chemistry, or biology. A model trained on Wikipedia might understand "who" or "where," but it often struggles with the "how" and "why" of scientific principles. This study addresses this gap by exploring whether domain-specific pre-training on academic literature can improve a model's ability to generate meaningful educational questions.
Methodology
The researchers developed EduQG by adapting the T5 (Text-to-Text Transfer Transformer) architecture through a three-stage pipeline designed to inject domain knowledge before task-specific training.
- Domain-Specific Pre-training: The foundational T5 model was further pre-trained on a subset of the S2ORC dataset, consisting of approximately 23.2 million scientific abstracts focused on Chemistry, Biology, and Physics.
- Fine-tuning: Following pre-training, the model was fine-tuned on SQuAD 1.1 to learn the mechanics of generating questions from a given context.
- Benchmarking: EduQG was compared against Leaf, a state-of-the-art baseline that fine-tunes T5 on SQuAD without the intermediate scientific pre-training step. Evaluation was conducted using the SciQ dataset, which contains over 13,000 crowdsourced scientific exam questions.
Key Results
The evaluation compared the models across predictive performance (how closely the generated question matches a human reference) and linguistic quality metrics.
Predictive Superiority
EduQG outperformed the Leaf baseline across all predictive metrics. It achieved a BLEU-1 score of 29.19 (vs. 27.07 for Leaf) and an F1-Score of 33.18 (vs. 30.90 for Leaf). These results indicate that domain knowledge gained from scientific abstracts significantly boosts the model's ability to predict "human-like" questions in a scientific context.
Linguistic Quality and Diversity
EduQG produced more diverse questions (0.749) compared to the baseline (0.735). However, a trade-off was observed in linguistic fluency. EduQG showed higher Perplexity (34.36 vs. 30.82) and a slightly higher Grammar Error rate (0.122 vs. 0.102). This suggests that while the model's content is more factually relevant, the dense and complex syntax of scientific abstracts may make the output less "conventional" according to standard linguistic evaluation models.
Discussion and Future Work
The study reinforces the idea that for specialized AI applications like EdTech, general-purpose datasets like Wikipedia are no longer sufficient benchmarks. The primary limitation identified is the mismatch in linguistic quality; scientific language is structurally different from the "standard" English used in linguistic evaluation models.
Future work will involve deeper linguistic analysis to understand why scientific pre-training affects fluency metrics and human evaluation to grade questions for pedagogical value rather than just automated similarity scores. The goal is to refine the pipeline to maintain domain expertise while preserving the grammatical polish of the base model.
Summary updated