Introduction
Effective teaching requires anticipating student inquiries to prepare supplemental materials or clarify complex points. However, educators often find it time-consuming to predict these questions manually. Automated Question Generation (QG) systems offer a potential solution but frequently suffer from "Wikipedia bias"—generating fact-based questions with explicit answers—or a "generality gap," where Large Language Models (LLMs) produce generic terminology-focused questions rather than addressing specific student confusion.
E-QGen is a pedagogical support system designed to bridge this gap. By generating realistic student questions based solely on a lecture abstract, E-QGen helps educators anticipate inquiries. It leverages multitask learning and a custom dataset of real-world YouTube student comments to produce nuanced, "student-like" questions that outperform general-purpose models like GPT-4.
Methodology
The E-QGen pipeline transforms a high-level lecture abstract into a suite of potential student inquiries through three main components:
- Educational Transcript Generator: Expands the user-provided abstract into a full, coherent lecture script.
- Reference Question Generator: Uses GPT-3.5 to generate general conceptual questions for broad topic coverage.
- Student Question Generator: The core innovation, utilizing a Vicuna-7b-v1.5 model fine-tuned via LoRA (Low-Rank Adaptation) and a multitask learning framework.
The multitask approach trains the model on three levels of data: actual questions ($P_G$) matched to transcripts, probable questions ($P_S$) aligned via semantic similarity, and potential questions ($P_P$) which are high-quality synthetic questions generated by GPT-4 to augment the training set.
Dataset Construction
To address data scarcity in educational AI, the authors curated a novel dataset by crawling 19,013 comments from 963 videos across 35 courses on MIT OpenCourseWare and Stanford Online. A "committee of LLMs" (PaLM 2, GPT-3.5, and GPT-4) identified actual questions, followed by manual verification. Questions were aligned to lecture segments using TextTiling for paragraph segmentation and cosine similarity for creating "golden" and "silver" pairs. GPT-4 was further used to generate 20 "platinum" questions per paragraph for effective fine-tuning.
Results and Evaluation
E-QGen was evaluated against GPT-3.5 and GPT-4 using standard NLP metrics, including ROUGE and BERTScore.
As shown in the results, E-QGen consistently outperformed GPT-4 across all metrics, achieving a ROUGE-L score of 0.2160 compared to GPT-4’s 0.1967. Qualitative analysis indicated that E-QGen’s questions aligned more closely with actual student concerns and provided a more diverse range of inquiries. Ablation studies further demonstrated that the specialized dataset and multitask fine-tuning were more critical than model scale, as the "SQ Only" model outperformed GPT-3.5.
Conclusion
E-QGen demonstrates the effectiveness of using synthetic data and multitask learning to specialize smaller models for niche educational tasks. While currently focused on Computer Science, future work aims to expand the system to other academic fields and further refine the alignment between student "confusion points" and lecture content.
Summary updated