Abstract: We present LearningQ, a challenging educational question generation dataset containing over 230K document-question pairs. It includes 7K instructor-designed questions assessing knowledge concepts being taught and 223K learner-generated questions seeking in-depth understanding of the taught concepts. We show that, compared to existing datasets that can be used to generate educational questions, LearningQ (i) covers a wide range of educational topics and (ii) contains long and cognitively demanding documents for which question generation requires reasoning over the relationships between sentences and paragraphs. As a result, a significant percentage of LearningQ questions (~30%) require higher-order cognitive skills to solve (such as applying, analyzing), in contrast to existing question-generation datasets that are designed mostly for the lowest cognitive skill level (i.e. remembering). To understand the effectiveness of existing question generation methods in producing educational questions, we evaluate both rule-based and deep neural network based methods on LearningQ. Extensive experiments show that state-of-the-art methods which perform well on existing datasets cannot generate useful educational questions. This implies that LearningQ is a challenging test bed for the generation of high-quality educational questions and worth further investigation.
Introduction
Automatic Question Generation (AQG) has the potential to revolutionize education by easing the burden on instructors and providing learners with endless practice opportunities. However, the field has been held back by its reliance on datasets like SQuAD or RACE. While large, these datasets were designed for reading comprehension, not education.
The authors argue that existing benchmarks are "shallow": the questions usually target low-level "remembering" (fact-finding) and can typically be answered by looking at a single sentence. In a real classroom, questions should challenge a student to apply, analyze, or evaluate concepts across a whole lesson. There was a distinct lack of a large-scale dataset that captured these higher-order cognitive demands—until LearningQ.
The LearningQ Dataset
The researchers introduced LearningQ, a massive new dataset of 230,000 document-question pairs designed to move AI beyond simple fact-retrieval toward true educational question generation. The data was distilled from two major platforms:
- TED-Ed: Providing 7K high-quality, instructor-designed questions.
- Khan Academy: Providing 223K questions asked by students during their learning process.
To ensure the learner-generated questions were actually useful, the authors trained a Convolutional Neural Network (CNN) to filter out "noise" (e.g., generic comments like "Thanks for the video!") from actual concept-relevant questions. This resulted in a dataset covering diverse subjects including Science, Math, and Humanities.
Quantitative and Qualitative Complexity
LearningQ differs significantly from previous benchmarks in both document length and cognitive complexity. While 100% of SQuAD questions fall into the lowest cognitive tier ("Remembering") of Bloom’s Revised Taxonomy, nearly 30% of LearningQ questions require higher-order skills like "Applying" or "Analyzing."
As shown in the statistical analysis, LearningQ documents are substantially longer. In SQuAD, 98% of questions are derived from a single sentence. In contrast, LearningQ questions are often related to an average of 3.5 (TED-Ed) to 6.6 (Khan Academy) sentences. This context gap means models cannot simply "transform" a single sentence into a question; they must synthesize information across paragraphs.
Furthermore, learner intent in LearningQ is distinct. Questions often start with "Why," indicating a search for deep causal relationships, whereas crowd-sourced datasets like SQuAD are dominated by "Who," "What," and "Where." Additionally, a significant portion of learner questions (~39% in Khan Academy) require external domain knowledge not explicitly stated in the text.
Benchmarking and Results
The authors tested three existing AQG approaches: a classic rule-based system (H&S), a standard neural encoder-decoder model (Seq2Seq), and a state-of-the-art Attention Seq2Seq model. The experiments revealed that LearningQ is a much more difficult benchmark:
- Performance Drop: Models that perform well on SQuAD (achieving Bleu-4 scores > 12) perform poorly on LearningQ (Bleu-4 scores < 4).
- The Context Challenge: There is a clear inverse correlation between the length of the source context and model performance. As the number of sentences required to generate a question increases, the models' ability to produce a coherent, relevant question declines sharply.
- Subject Sensitivity: Models performed slightly better on "structured" subjects like Math and Science compared to Humanities, likely due to more consistent terminology.
Conclusion and Future Work
LearningQ proves that current state-of-the-art models struggle significantly when questions require reasoning across entire documents rather than single sentences. Future research must focus on "document-to-question" models that can handle long-range dependencies and incorporate external knowledge graphs to bridge the gap in educational reasoning.
Summary updated