TL;DR
This paper introduces a novel neural question generation (QG) method that can produce question-answer pairs tailored to specific difficulty levels, estimated using Item Response Theory (IRT). This advancement is crucial for educational applications like intelligent tutoring systems, enabling more personalized and effective learning experiences by matching question difficulty to learner ability.
What Problem Does This Paper Address?
Generating questions for reading comprehension is a valuable tool for education, helping students focus and identify misconceptions. However, existing QG methods often struggle to control the difficulty of the generated questions, making them less effective for adaptive learning systems. Furthermore, many current methods only generate questions, not answers, which prevents automatic scoring and adaptive difficulty adjustment. This paper tackles the challenge of creating QG systems that not only generate relevant question-answer pairs but also precisely control their difficulty, aligning with a learner's estimated ability.
Methods
The core of this work lies in a two-stage approach that leverages pre-trained transformer models:
- Difficulty-Controllable Answer Extraction (BERT-based): This stage uses a fine-tuned BERT model to extract an answer text from a given passage. The novelty here is the incorporation of a target difficulty value as input, allowing the model to learn to extract answers that correspond to a specific difficulty level.
- Difficulty-Controllable Answer-Aware Question Generation (GPT-2-based): This stage uses a fine-tuned GPT-2 model to generate a question given a reading passage and a target answer. Crucially, it also takes the target difficulty value as input, enabling it to generate questions whose difficulty is influenced by this parameter.
The difficulty itself is quantified using Item Response Theory (IRT), a statistical framework that models the relationship between an individual's ability and their performance on test items (questions). Specifically, the paper utilizes the Rasch model, a one-parameter logistic model, to estimate question difficulty.
Datasets
- SQuAD (Stanford Question Answering Dataset): This dataset, consisting of reading passages and question-answer pairs, was used as the foundation.
- Extended SQuAD Dataset: The authors created a new dataset by augmenting SQuAD with IRT-based difficulty values. This was achieved by training various QA models to generate answers for SQuAD questions, then using these answers to estimate question difficulty via IRT. This extended dataset contains quadruplets of (reading passage, question text, answer text, IRT-based difficulty).
Key Insights and Ideas
- IRT as a Difficulty Quantifier for QG: Applying IRT to quantify question difficulty for QG is a novel approach. It provides a principled way to measure how hard a question-answer pair is, moving beyond simple heuristics.
- Difficulty is a Property of the Pair: The paper emphasizes that difficulty is not just about the question but also the answer. Their method generates both, allowing for a more holistic control of difficulty.
- Transformer Models Adaptable to Difficulty: The success of fine-tuning BERT and GPT-2 with difficulty inputs demonstrates the flexibility of these powerful language models for tasks beyond standard generation, incorporating nuanced control parameters.
- QA Systems as Proxies for Learners: Using automated QA systems to generate response data for IRT estimation is a practical workaround for the cost and complexity of collecting human responses, though it raises questions about generalizability.
Key Results
- Human Evaluation: Human judges rated generated questions with higher difficulty values as more difficult, indicating the method's ability to control difficulty.
- Fluency and Relevance: Over 90% of generated questions were grammatically correct or acceptable and relevant to the reading passages.
- Answerability: Approximately 70% of generated question-answer pairs were fully answerable, with about 85% being partially answerable.
- Answer Length Correlation: The average word length of generated answers increased with target difficulty, suggesting that longer answers are associated with higher difficulty, which aligns with intuition.
Limitations and Future Work
- SQuAD Dataset Dependency: The reliance on SQuAD, which has been criticized for its focus on surface-level similarity rather than deep reasoning, is a limitation. Future work should explore other datasets.
- Scale of Human Evaluation: The human evaluation was conducted on a relatively small sample (60 pairs), limiting the reliability of the results. Larger-scale evaluations are planned.
- QA System Substitution: The paper acknowledges the need to further investigate whether QA systems can truly substitute for human learners in difficulty estimation. A comparison of IRT-based difficulties from QA systems versus human learners is suggested.
- Adaptive QG Systems: The authors propose extending their method to build full-fledged computerized adaptive testing (CAT) systems, which would dynamically adjust question difficulty based on learner performance.
- Finer Granularity of Difficulty: Using a larger and more diverse set of QA systems could lead to more accurate and finely-grained difficulty estimations.
Summary updated