Pre-Training With Scientific Text Improves Educational Question Generation

Hamze Muse, Sahan Bulathwela, Emine Yilmaz

Abstract

With the boom of digital educational materials and scalable e-learning systems, the potential for realising AI-assisted personalised learning has skyrocketed. In this landscape, the automatic generation of educational questions will play a key role, enabling scalable self-assessment when a global population is manoeuvring their personalised learning journeys. We develop EduQG, a novel educational question generation model built by adapting a large language model. Our initial experiments demonstrate that EduQG can produce superior educational questions by pre-training on scientific text.

Introduction

As digital e-learning scales globally, the demand for "self-assessment" tools—automated systems that generate questions to help students verify their mastery—has increased significantly. However, manual question creation remains a labor-intensive bottleneck. While automated Question Generation (QG) has advanced, most state-of-the-art models are trained on general-purpose datasets like SQuAD 1.1. While effective for general reading comprehension (e.g., "Who is the president of France?"), these models often lack the rigor and specialized vocabulary required for STEM subjects. For AI to serve as an effective tutor, it must understand the nuances of scientific domains like physics, chemistry, and biology.

Methodology

To address this gap, we developed EduQG, a model designed to leverage domain-specific knowledge for educational purposes. The architecture is based on a transfer learning approach using the T5 (Text-to-Text Transfer Transformer) as a foundation.

Unlike standard baselines, EduQG undergoes a three-stage process:

  1. Domain Pre-training: The model is first pre-trained on a subset of the S2ORC (Semantic Scholar Open Research Corpus), containing 23.2 million scientific abstracts. This stage allows the model to internalize scientific terminology and logical structures.
  2. Fine-tuning: The model is then fine-tuned on SQuAD 1.1 to learn the mechanics of generating questions from a given context.
  3. Evaluation: Finally, the model is evaluated on SciQ, a dataset of crowdsourced scientific exam questions, to test its performance on real-world educational content.
The training and evaluation pipeline for EduQG compared to the Leaf baseline.

We compared EduQG against Leaf, a state-of-the-art QG system that fine-tunes T5 on SQuAD but omits the scientific pre-training step.

Key Results

Our experiments demonstrate that domain-specific pre-training significantly enhances the model's ability to generate relevant scientific questions. EduQG outperformed the Leaf baseline across all predictive performance metrics.

Comparative performance of Leaf and EduQG across predictive and linguistic metrics.

Predictive Performance

EduQG showed a marked improvement in matching human-generated questions:

  • BLEU-1 Score: EduQG achieved 29.19, compared to Leaf’s 27.07.
  • F1-Score: EduQG reached 33.18, outperforming the baseline's 30.90.
  • Consistent gains were observed across BLEU-2, BLEU-3, and BLEU-4 metrics, proving that the model better captures the context of scientific text.

Linguistic Quality and Diversity

EduQG produced a wider variety of question types, as evidenced by a higher Diversity score (0.749 vs 0.735). However, we observed a "Linguistic Quality Paradox": EduQG had slightly higher Perplexity (34.36 vs 30.82) and a marginally higher Grammar Error rate (0.122 vs 0.102). We hypothesize that scientific language is inherently more complex and dense than the Wikipedia-style English used to calibrate these standard metrics, leading to higher perplexity scores despite higher content relevance.

Discussion and Future Work

The core insight of this research is that domain knowledge is more critical than general fluency for educational tools. While a model might be grammatically perfect, it is of little use if it cannot grasp scientific relationships.

However, the limitations of automated metrics suggest that future work must prioritize human evaluation. Teachers and students should rate these questions to determine if the increased scientific relevance translates to better learning outcomes. Additionally, we aim to move beyond benchmarking and integrate these models into live e-learning environments to test their real-world utility.

Summary updated

Abstract

With the boom of digital educational materials and scalable e-learning systems, the potential for realising AI-assisted personalised learning has skyrocketed. In this landscape, the automatic generation of educational questions will play a key role, enabling scalable self-assessment when a global population is manoeuvring their personalised learning journeys. We develop EduQG, a novel educational question generation model built by adapting a large language model. Our initial experiments demonstrate that EduQG can produce superior educational questions by pre-training on scientific text.
BibTeX
@inproceedings{muse2023pre,
  author = {Hamze Muse and Sahan Bulathwela and Emine Yilmaz},
  title = {{Pre-Training With Scientific Text Improves Educational Question Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=aK8ovw52po},
}
Figure