TL;DR
This paper tackles the challenge of automatically generating higher-order reading comprehension questions, which are crucial for effective learning but scarce in existing datasets. They demonstrate that few-shot prompting of Large Language Models (LLMs) can significantly outperform traditional neural models in generating these valuable questions, even with minimal training data.
What Problem Does This Paper Address?
Reading comprehension is a cornerstone of education, and effective learning often relies on asking insightful questions. However, the internationally recognized PIRLS (Progress in International Reading Literacy Study) standards emphasize that good comprehension questions should go beyond simple information retrieval, requiring higher-order thinking skills like inferencing, integrating, and evaluating. Unfortunately, publicly available datasets for training automatic question generation (QG) models are heavily skewed towards simpler, factoid questions, leaving a significant gap in the ability to generate these more complex, higher-order questions. This scarcity hinders the development of tools that can truly support deeper learning and assessment, especially in languages other than English.
Methods
The authors investigate the use of Large Language Models (LLMs) for few-shot question generation, meaning they provide the LLM with only a few examples to guide its output. They compare this approach against two established neural QG models: a pipeline model and a seq2seq model.
- LLM Approach (Zero-shot and Few-shot): They utilize a Chinese version of Stanford Alpaca, a LLaMA-based model. For zero-shot prompting, they provide a prompt instructing the model to generate a question requiring deduction. For few-shot prompting, they augment this prompt with five example passage-question pairs. They experimented with various prompts and found that a prompt focusing on deduction and requiring non-word-matching questions yielded the best results.
- Baseline Models:
- Pipeline Model: This model first generates an answer using an extractor and then uses a base model fine-tuned with UNIMO for question generation.
- Seq2seq Model: This model is trained end-to-end to generate question-answer pairs directly from a passage.
Datasets
The evaluation data was drawn from the development set of DuReader_robust, a widely used Chinese Q&A dataset. The authors note that the pipeline model might not perform optimally due to its filtering step. For their few-shot LLM experiments, they used five sample passage-question pairs derived from public Chinese-language examination papers.
Key Insights and Ideas
- LLMs Excel at Higher-Order Questions: The core finding is that LLMs, even with minimal supervision (few-shot learning), are significantly better at generating higher-order questions compared to traditional neural QG models. This suggests LLMs have a more nuanced understanding of language and reasoning required for complex question formulation.
- Few-Shot Learning Boosts Performance: The performance of the LLM improved noticeably when moving from zero-shot to few-shot prompting, highlighting the effectiveness of providing even a small number of examples.
- PIRLS as a Standard for Evaluation: This paper is the first to evaluate QG models specifically against the PIRLS standards for reading comprehension, providing a more educationally relevant benchmark.
- Human Agreement is High: The study demonstrates a high level of agreement among human assessors on both the usability and the PIRLS question type classification, validating their evaluation methodology.
Key Results
- Usability: LLM-based approaches (zero-shot and few-shot) achieved significantly higher usability rates (62.5% and 71% usable without revision, respectively) compared to the pipeline (35% usable without revision) and seq2seq (35% usable without revision) models.
- Higher-Order Questions: Few-shot LLM prompting generated a substantially higher proportion of higher-order questions (31.5%) compared to zero-shot LLM (29.5%) and the neural models (which produced very limited numbers).
- Agreement: Human assessors showed substantial agreement on usability (Kappa=0.739) and almost perfect agreement on question types (Kappa=0.861).
Limitations and Future Work
- Pedagogical Impact: The evaluation focused on question quality but did not assess their pedagogical impact on students. The authors emphasize that automatically generated questions should be treated as drafts and require teacher review.
- Tailored Prompts: The authors plan to investigate more tailored prompts for producing specific PIRLS question types.
- Chinese Higher-Order Dataset: They aim to construct a dedicated Chinese dataset of higher-order questions for fine-tuning LLMs, which could further improve performance.
Summary updated