Abstract
Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, Teruko Mitamura. Proceedings of COLING 2016.
Researchers at Carnegie Mellon University have developed a system that generates multiple-choice questions requiring "deep" reading comprehension by linking information across multiple sentences. By leveraging semantic analysis—specifically coreference resolution and paraphrase detection—the system moves beyond simple surface-level pattern matching to create questions that truly test a learner's ability to connect the dots.
Introduction: The "Shallow" Question Problem
Most automated question generation (QG) systems are "shallow." They typically take a single sentence and apply syntactic transformations to turn it into a question (e.g., "The cat sat on the mat" becomes "Where did the cat sit?"). While these systems produce grammatically correct questions, they don't actually measure reading comprehension. A student can often answer them just by finding the sentence with the same keywords.
This paper addresses the need for higher-level questions—those that require "inference steps" across a text. To answer these, a reader must understand that a pronoun in sentence two refers to a complex process mentioned in sentence one, or recognize that two different phrases are actually describing the same event.
System Architecture
The proposed system follows a pipeline that starts with text and ends with multiple-choice questions, incorporating target selection, question construction, and distractor generation.
Methodology
The authors propose a two-pronged approach to question generation, complemented by a novel distractor generation strategy.
QG1: Leveraging Semantic Glue
This system uses templates to find "semantic glue" between sentences. It looks for event coreference (e.g., "divide" in one sentence and "division" in another) and entity coreference (e.g., "a few tumor cells" and "they"). By identifying these links, the system can ask a question using information from one sentence where the answer resides in another.
QG2: Concept Relationships
This is a more flexible, pattern-based system. It extracts generic patterns from existing human-generated Q&A pairs (like "What happens because of [Entity] [Trigger]?") and attempts to apply them to new passages based on the relationships between events and entities.
Distractor Generation
To create a multiple-choice question, plausible wrong answers are required. The system builds an event graph of the entire passage. It identifies the "nodes" (events) used in the question and the correct answer, then selects other events from the graph to turn into phrase-level distractors. This ensures the distractors are contextually relevant but factually incorrect for that specific question.
Evaluation
The study utilizes the ProcessBank corpus, which consists of 200 paragraphs about biological processes extracted from a high school biology textbook. Biology is particularly suitable as it is heavy on causal chains and complex sequences. ProcessBank includes high-quality human annotations for event triggers, entity mentions, coreference links, and causal relations, allowing the researchers to focus on the logic of question generation.
The researchers compared their systems against a well-known baseline (Heilman and Smith, 2010):
- Inference Steps: QG1 achieved an average of 0.76 inference steps per question, significantly higher than the baseline's 0.16.
- Grammaticality: Despite the increased complexity, QG1 maintained a grammatical correctness score (1.50 on a 1-3 scale, where 1 is best) comparable to the baseline (1.34).
- Distractor Quality: Human evaluators rated the distractors around 1.94 on a 3-point scale, indicating they were generally viable but incorrect.
Conclusion
The authors demonstrate that coreference resolution is the key to breaking the "single-sentence barrier" in automated QG. By linking mentions across sentences, the system forces the learner to maintain a mental model of the text. While currently reliant on expert annotations, future work aims to integrate automated semantic parsers to create a fully end-to-end system capable of generating deep questions from raw text.
Summary updated