TL;DR
This paper introduces a novel multi-task learning approach to improve answer-aware question generation. By incorporating language modeling as an auxiliary task, their model learns richer representations, leading to more coherent and fluent questions. This method achieves state-of-the-art results on benchmark datasets and demonstrates improved question quality through human evaluation.
What Problem Does This Paper Address?
The paper tackles the challenge of answer-aware question generation (QG). Given a piece of text and a specific answer span within it, the goal is to generate a question for which that answer is the correct response. While QG has numerous applications, such as aiding question answering systems, making dialogue systems more proactive, and supporting educational tools, generating high-quality, relevant, and grammatically correct questions remains difficult. Existing neural approaches, primarily based on encoder-decoder architectures with attention mechanisms, have made progress but still have room for improvement in terms of question fluency and coherence.
Methods
The authors build upon the attention-based pointer-generator model, a standard architecture for sequence-to-sequence generation tasks. Their core innovation is the introduction of multi-task learning, specifically by adding language modeling (LM) as an auxiliary task.
Here's a breakdown of their approach:
- Baseline Model: A feature-enriched pointer-generator network. This model uses a bidirectional LSTM encoder to process input text, answer position, and lexical features (like POS tags, NER tags, and word case). A unidirectional LSTM decoder, coupled with an attention mechanism, generates the question. The pointer-generator mechanism allows the model to both generate words from a vocabulary and copy words directly from the source text.
- Auxiliary Task: Language Modeling: The LM task is designed to predict the next word and the previous word in a sequence. This is achieved using a bidirectional LSTM that processes the input text.
- Hierarchical Multi-Task Learning: The key to their approach is how these two tasks are combined. They propose a hierarchical structure where the language modeling task acts as a "low-level" task. The hidden states from the LM are fed into the main QG encoder, enriching its representation of the input text. This is intended to help the encoder learn better representations that benefit the decoder in generating more coherent questions.
- Combined Objective: The overall training objective is a weighted sum of the QG loss (E) and the LM loss (Elm), controlled by a hyperparameter β.
Datasets
The authors evaluate their model on two widely used reading comprehension datasets:
- SQuAD (Stanford Question Answering Dataset): Used for training, development, and testing.
- MARCO (Microsoft MAchine Reading COmprehension): Also used for training, development, and testing.
They utilize existing data splits and pre-processed lexical features from previous work.
Key Insights and Ideas
- Language Modeling as a "Free Lunch" for QG: The paper posits that language modeling, a fundamental NLP task, can significantly boost the performance of question generation without requiring additional human annotations. This is because LM inherently encourages the model to learn richer linguistic patterns and semantic coherence.
- Hierarchical Structure Matters: The authors emphasize that a hierarchical combination of tasks, where LM informs the QG encoder, is more effective than simply running tasks in parallel or at the same level. This suggests that the way auxiliary tasks are integrated can be crucial for their effectiveness.
- Robustness to Lexical Features: The results show that even when lexical features are removed, the language modeling task still provides a substantial performance boost. This indicates that their approach is robust and can be applied to scenarios where high-quality lexical features are not readily available, potentially making it useful for low-resource languages.
- Beyond BLEU: The paper highlights the importance of using metrics like perplexity and distinctness, in addition to BLEU, to evaluate the fluency and diversity of generated questions, as BLEU alone might not capture all aspects of generation quality.
Key Results
- State-of-the-Art Performance: The proposed multi-task learning model, with both features and language modeling, achieves state-of-the-art results on both SQuAD and MARCO datasets, reaching a BLEU-4 score of 16.23 on SQuAD and 20.88 on MARCO.
- Significant Gains from LM: The language modeling auxiliary task consistently yields noticeable performance improvements across all evaluation metrics (BLEU, perplexity, and distinctness) compared to the baseline model.
- Human Evaluation Success: Human evaluation confirms the superiority of their model. Generated questions received higher scores for matching, fluency, and relevance compared to the baseline.
- Perplexity and Diversity Improvements: The model with language modeling generates questions with lower perplexity (indicating better fluency) and higher distinctness (indicating better diversity).
Limitations and Future Work
- Generalization of LM: The authors suggest that in future work, they will explore applying the auxiliary language modeling task to other neural generation systems to test its generalization ability.
- Hyperparameter Tuning: While they found a good value for β (0.6) through grid search, the optimal weighting of tasks can be a complex area for further research.
- Deeper Encoder vs. LM: The paper shows that simply adding a deeper encoder layer is not as effective as incorporating the language modeling task, suggesting that the nature of the auxiliary task is more important than just increasing model complexity.
Summary updated