TL;DR
This paper tackles the challenge of automatically generating educational questions from text by first overgenerating potential questions using rule-based transformations and then ranking them with a statistical model. This "overgenerate-and-rank" approach significantly improves the quality of generated questions, nearly doubling the acceptance rate of the top-ranked outputs.
What Problem Does This Paper Address?
Creating effective educational materials, particularly for reading practice and assessment, often requires generating relevant questions about a given text. Manually crafting these questions is time-consuming. While automatic question generation (QG) systems exist, they often struggle with producing high-quality, fact-based questions that are grammatically correct, semantically meaningful, and not trivially obvious. Existing rule-based systems can generate many questions, but filtering the good from the bad is a significant hurdle. This paper aims to bridge this gap by developing a method to automatically generate and then effectively rank these questions, making them more useful for educational purposes.
Methods
The authors propose a two-step approach:
- Rule-based Overgeneration:
- Sentence Simplification: The system first simplifies declarative sentences from the source text. This involves altering lexical items, syntactic structures, and semantics. For example, it might remove leading conjunctions, sentence-level modifying phrases, or appositives. It also extracts declarative sentences from finite clauses, relative clauses, and participial phrases to capture embedded information.
- Question Transformation: The simplified sentences are then transformed into questions using a sequence of general-purpose syntactic and lexical rules. This includes common transformations like subject-auxiliary inversion and WH-movement. The system identifies potential "answer phrases" (like noun phrases or prepositional phrases) and converts them into "question phrases" (containing WH-words). Crucially, they incorporate linguistic knowledge to handle complex phenomena like long-distance dependencies and "island constraints" (e.g., noun phrase islands) that govern WH-movement.
- Statistical Ranking:
- A logistic regression model is trained to rank the overgenerated questions. This model learns to predict the acceptability of a question based on a rich set of features. These features include:
- Length Features: Number of tokens in the question, source sentence, and answer phrase, along with histogram features.
- WH Words: Presence of specific WH words (who, what, where, etc.).
- Negation: Presence of negation words.
- N-Gram Language Model Features: Likelihoods from unigram and trigram models for the question and source sentence.
- Grammatical Features: Counts of various grammatical categories (proper nouns, pronouns, phrases, clauses) derived from parse trees.
- Transformations: Binary features indicating which rule-based transformations were applied.
- Vagueness: Features related to potentially vague noun phrases.
- Histograms: Binary features indicating if counts or lengths exceed certain thresholds.
Datasets
The paper describes the acquisition of a dataset for training and evaluating their question ranking model.
- Training and Test Data: Approximately 2,807 training questions and 428 test questions were collected.
- Corpora: The data was generated from three sources:
- English Wikipedia: Featured articles (250-2000 words), providing expository texts at an adult reading level.
- Simple English Wikipedia: Articles of similar length, offering text at an elementary education or intermediate second language learning level.
- Wall Street Journal (Penn Treebank): Section 23 of the WSJ corpus, a standard benchmark for NLP tasks.
- Annotation: University students rated questions for acceptability, identifying various deficiencies (ungrammaticality, vagueness, obvious answers, missing answers, etc.). For the test set, questions were rated by three annotators, and a question was deemed acceptable only if a majority agreed.
Key Insights and Ideas
- Overgenerate-and-Rank is Effective: The core idea that generating many candidates and then using a statistical model to select the best ones is a powerful strategy for QG, mirroring successful applications in other NLP areas like parsing and generation.
- Linguistic Rules + Statistical Ranking Synergy: Combining explicit linguistic knowledge (encoded in rules for transformations) with statistical learning (for ranking) is crucial. Rules capture complex grammatical phenomena that are hard to learn from limited data, while the ranker can capture subtle quality indicators not easily encoded in rules.
- Beyond Surface Features: The study demonstrates that incorporating features beyond simple surface characteristics (like sentence length) is vital for effective ranking. Grammatical features and transformation indicators proved particularly important.
- Vagueness and Nonsense are Key Issues: The ranking model appears to effectively target common deficiencies like nonsensical or vague questions, significantly reducing their prevalence in the top-ranked outputs.
Key Results
- Doubled Acceptability: The most significant result is that ranking nearly doubled the percentage of acceptable questions. For the unranked, rule-based overgenerated questions, only 27.3% were rated acceptable. By restricting to the top 20% of ranked questions, this acceptability rate rose to 52.3%.
- Deficiency Reduction: Ranking significantly reduced the occurrence of specific deficiencies. "Does not make sense" questions dropped from 20.6% to 4.7%, and "Vague" questions dropped from 19.6% to 7.0%. Ungrammaticality also decreased from 14.0% to 10.5%.
- Feature Importance: Grammatical features were found to be the most impactful, with their removal causing a substantial drop in acceptability.
Limitations and Future Work
- Inter-rater Agreement: The inter-rater agreement (Fleiss's κ = 0.42) was described as "moderate," suggesting some subjectivity in acceptability judgments.
- Small Training Data: The ranking model was trained on a "small, tailored dataset." The authors speculate that larger, less noisy training sets might improve the performance of features like N-gram language models.
- Future Work: The authors mention that generating correct answers and distractors for questions is left for future work. They also suggest that incorporating paraphrase generation techniques could enhance their approach. The system could also be extended to generate other question types (e.g., "how," "why," "what kind of").
Summary updated