Abstract
Neural models with an encoder-decoder framework provide a feasible solution to Question Generation (QG). However, after analyzing the model vocabulary we find that current models (both RNN-based and pre-training based) have more than 23% inflected forms. As a result, the encoder will generate separate embeddings for the inflected forms, leading to a waste of training data and parameters. Even worse, in decoding these models are vulnerable to irrelevant noise and they suffer from high computational costs. In this paper, we propose an approach to enhance the performance of QG by fusing word transformation. Firstly, we identify the inflected forms of words from the input of encoder, and replace them with the root words, letting the encoder pay more attention to the repetitive root words. Secondly, we propose to adapt QG as a combination of the following actions in the encode-decoder framework: generating a question word, copying a word from the source sequence or generating a word transformation type. Such extension can greatly decrease the size of predicted words in the decoder as well as noise. We apply our approach to a typical RNN-based model and UniLM to get the improved versions. We conduct extensive experiments on SQuAD and MS MARCO datasets. The experimental results show that the improved versions can significantly outperform the corresponding baselines in terms of BLEU, ROUGE-L and METEOR as well as time cost.
Introduction
In the field of Natural Language Processing (NLP), Question Generation (QG) aims to automatically create questions based on a given text snippet and a target answer. While modern QG models typically utilize an encoder-decoder architecture, they suffer from a significant yet overlooked inefficiency: vocabulary redundancy.
Analysis of state-of-the-art models reveals that over 23% of the vocabulary consists of inflected forms (e.g., "run," "running," "ran"). Because these are treated as distinct entities, the encoder must learn separate embeddings for words sharing the same semantic root, leading to wasted parameters. Furthermore, a bloated output vocabulary introduces decoding noise and increases computational latency during inference.
Methodology
To address these issues, we introduce a Word Transformation (WT) approach designed to be integrated into existing architectures like the RNN-based Pointer Generator Network or the Transformer-based UniLM.
Root-Based Encoder
The first stage involves preprocessing the input sequence to identify inflected words and replace them with their root forms (lemmas). By collapsing variants into a single root, the encoder is forced to focus on core semantic meanings rather than surface-level morphological variations.
Three-Action Decoder
Rather than selecting from a massive vocabulary of tens of thousands of words, our decoder employs a "three-way soft switch" to determine the next token based on three possible actions:
- Question Word Generation: Selecting from a small, specialized vocabulary (~1,000 words) of common question-related terms (e.g., "what", "how").
- Word Copying: Directly copying a root word from the source input.
- Transformation Type Generation: Applying a specific rule to modify a copied or generated word.
This mechanism allows the model to generate complex forms like "did" by selecting the root "do" and applying a past-tense transformation. We define nine specific transformation types to cover common English inflections across various parts of speech.
Evaluation and Results
We evaluated the WT-enhanced models on the SQuAD and MS MARCO benchmarks. The results demonstrate that reducing the decoder's search space leads to significant performance gains:
- Accuracy: On MS MARCO, the Pointer+WT model improved BLEU4 scores from 17.90 to 22.10. Similar significant gains were observed across BLEU, ROUGE-L, and METEOR metrics for both RNN and UniLM baselines.
- Efficiency: The approach drastically reduced decoding time. The RNN-based model achieved a 39% reduction in time cost, while the UniLM-based model saw a 13% improvement.
- Human Preference: In blind testing, human annotators preferred questions from the WT-enhanced model over the baseline by a ratio of roughly 2:1 (73 vs 36 cases out of 200).
Conclusion
By treating inflected forms as redundant and utilizing a specialized three-action decoder, we can significantly improve both the accuracy and efficiency of Question Generation models. While current implementations rely on external linguistic tools for lemmatization, future work will explore learning these transformations end-to-end to improve robustness across different languages.
Summary updated