Abstract
Automatic Question Generation (QG) is a critical component for educational tools and conversational agents. However, the output of these systems often requires manual curation. Traditionally, this human effort is treated as a final post-processing step, and the underlying system remains static. This paper presents GEN, a pattern-based QG system designed to learn from human interaction. By treating user edits as implicit feedback, GEN automatically acquires new generation patterns and refines its ranking logic. Experimental results show that this "expert-in-the-loop" approach leads to performance improvements of over 10% across standard NLP metrics.
Introduction
While neural sequence-to-sequence models have advanced the state-of-the-art in Question Generation, they often struggle with online learning—the ability to immediately incorporate a single human correction to improve future outputs. In practical applications, human experts (e.g., teachers) must often edit or discard generated questions. The GEN system aims to capture this "wasted effort" and turn it into a high-quality stream of training data, allowing the system to evolve incrementally.
The GEN System Architecture
GEN utilizes a pattern-based architecture that is more transparent and agile than black-box neural models. The workflow is divided into three primary modules: Question Generation, Pattern Acquisition, and Pattern Improvement.
- Pattern Acquisition: The system begins with a small set of "seed" pairs. It uses Semantic Role Labeling (SRL) and dependency parsing to identify the underlying structure of sentences.
- Question Generation: New sentences are matched against a library of patterns. If a match is found, the sentence is transformed into a question.
- Implicit Feedback Loop: This is the core innovation. When a user corrects a question, the corrected pair becomes a new seed for pattern discovery. Simultaneously, the system tracks which patterns produce questions that require the least editing, weighting them more heavily in future rankings.
Sentence Annotation and Pattern Discovery
To create generalized patterns, GEN performs deep linguistic analysis on source sentences. This involves generating constituency trees, dependency trees, and identifying semantic roles (e.g., Agent, Patient).
By aligning these structures between a sentence and its corresponding question using the Hungarian algorithm, GEN can extract abstract transformation rules. These rules are further generalized using Lexical, WordNet, and Word2Vec features to ensure they can be applied to diverse contexts.
Experimental Evaluation
The system was evaluated using two datasets: MONSERRATE, a specialized corpus for iterative learning, and SQuAD, a large-scale benchmark.
The results demonstrated that incorporating implicit feedback consistently improved performance. GEN achieved gains of 10% or more in ROUGE, METEOR, and BLEU scores compared to a static version of the system. Notably, GEN outperformed a standard neural QG baseline in specific configurations, particularly when focusing on the top-ranked questions. The system proved efficient, showing significant learning even with small batches of feedback (7-12 sentences).
Conclusion
The GEN system demonstrates that human curation can be a powerful driver for system improvement rather than just a final filter. By leveraging a pattern-based architecture and an implicit feedback loop, GEN provides an explainable and adaptive solution for Question Generation that effectively bridges the gap between automated systems and human expertise.
Summary updated