TL;DR
This paper tackles the challenge of generating grammatically correct and semantically relevant questions from text and answers. They introduce a novel "question-type driven" approach that first predicts the question type (e.g., "who," "what," "when") and then uses this prediction to guide the question generation process, significantly improving accuracy and achieving state-of-the-art results.
What Problem Does This Paper Address?
Generating questions from a given text and a specific answer is a crucial task for applications like question answering systems, dialogue agents, and educational tools. While existing neural network-based methods (like seq2seq models with attention) have shown promise, they often struggle with a fundamental issue: generating questions with the correct type. For instance, a model might generate a "how" question when the answer is a person's name, leading to an irrelevant or nonsensical question. This mismatch between the predicted question type and the actual answer is a significant hurdle that this paper aims to overcome.
Methods
The core innovation of this paper is a unified model that simultaneously predicts the question type and generates the question.
- Feature-Rich Encoder: The model starts with an encoder that processes the input sentence and the answer span. It enriches the word embeddings with lexical features (POS tags, NER tags, word case) and answer position embeddings. A bidirectional LSTM then generates a sequence of hidden states.
- Question Type Prediction: An auxiliary unidirectional LSTM network is employed to predict the question type. It takes the hidden states corresponding to the answer span as input. This prediction is crucial because the question type dictates the question's structure and phrasing.
- Unified Attention-based Decoder: The decoder is a standard seq2seq architecture but with a key modification: instead of starting with a generic
<BOS>token, it uses the predicted question word (e.g., "What") as the initial input. This directly injects the question type information into the generation process. Furthermore, the decoder's context vector is conditioned not only on the encoder's hidden states but also on the question type prediction's hidden state. - Combined Training Objective: The model is trained with a combined loss function that includes both the question generation loss and the question type prediction loss, encouraging the model to optimize both tasks jointly.
Datasets
The experiments were conducted on two widely used reading comprehension datasets:
- SQuAD (Stanford Question Answering Dataset): A large dataset of questions, answers, and context passages.
- MARCO: Another large-scale dataset for question answering.
Key Insights and Ideas
- Question Type is King: The paper strongly emphasizes that accurately predicting the question type is vital for generating coherent and relevant questions. An incorrect type can lead the entire generation process astray.
- Unified Approach Benefits Both Tasks: By jointly training question type prediction and question generation, the model not only improves question generation but also significantly boosts the accuracy of question type prediction itself.
- Leveraging Answer Context: The model's ability to use the answer's hidden states and surrounding context for type prediction is key to resolving ambiguities and generating more precise questions.
Key Results
- Question Type Accuracy: The proposed unified model achieved significant improvements in question type prediction accuracy, with a 16.79% gain on SQuAD and a 3.5% gain on MARCO compared to a baseline.
- State-of-the-Art Question Generation: The model achieved new state-of-the-art results on both SQuAD (BLEU-4 of 16.31) and MARCO (BLEU-4 of 21.59) datasets.
- Upper Bound Analysis: Experiments showed that even simply providing the correct question word as input (without the full type prediction mechanism) led to substantial improvements, highlighting the importance of this information.
Limitations and Future Work
- Unbalanced Question Types: The paper notes that the distribution of question types in datasets like SQuAD is unbalanced, making accurate prediction challenging.
- Future Directions: The authors suggest that their approach provides a promising direction for question generation, implying that further refinements and applications of this question-type driven methodology are possible. The paper also implicitly points towards the need for more balanced datasets or advanced techniques to handle imbalanced class distributions in question type prediction.
Summary updated