Introduction
Question Generation (QG) is the task of automatically creating a question based on a given passage and a specific answer. This task is inherently "one-to-many," as a single text segment can support a multitude of valid, high-quality questions. Traditional approaches to Reinforcement Learning for Question Generation (RLQG) often struggle with two primary challenges: sample inefficiency and a lack of diversity.
Standard RL agents often fail to explore the vast space of possible word combinations effectively, leading to slow learning. Furthermore, algorithms like REINFORCE are typically designed for reward maximization, which can lead to "mode collapse"—where the model repeatedly generates a single "safe" question format while ignoring other valid alternatives. To address these issues, this paper introduces BPD-DSRL, a framework combining Bilevel Policy Decomposition with a Diversity-Seeking Reinforcement Learning objective.
Methodology
The BPD-DSRL framework decomposes the complex QG task into more manageable sub-tasks and employs a novel objective to maintain output variety.
Bilevel Policy Decomposition (BPD)
Instead of generating a question directly from the source text, BPD-DSRL utilizes a two-step process:
- Rationale Policy: The model first generates an "answer-centric summary" or rationale. This rationale acts as a bridge, distilling the relevant context from the source passage.
- Action Policy: The model then uses this rationale as a concise "working memory" to generate the final question.
This "divide and conquer" strategy significantly reduces the search space for each step, enhancing sample efficiency and making the source-to-target mapping less ambiguous.
Diversity-Seeking RL (DSRL)
To combat mode collapse, the authors implement a DSRL objective inspired by GFlowNets and Soft Q-learning. Rather than seeking only the single highest reward, the goal is to make the probability of generating a specific question proportional to its reward. This "reward matching" approach ensures that if multiple valid questions exist for a passage, the model learns to sample from all of them rather than converging on a single repetitive formula.
Reward Modeling
The system is guided by two distinct reward signals:
- Process Reward: An intermediate reward using a Natural Language Inference (NLI) model to ensure the generated rationale is logically supported by the original passage.
- Outcome Reward: A final reward that evaluates the generated question based on its linguistic fluency and "answerability"—the ability of a separate reading comprehension model to find the correct answer using the generated question.
Experimental Evaluation
The researchers evaluated BPD-DSRL on major benchmarks, including SQuAD 1.1 and NewsQA. The model, based on a T5-large architecture, was compared against standard RL baselines and large language models (LLMs).
Quality and Diversity
BPD-DSRL consistently outperformed existing baselines (such as PPO and REINFORCE) in both Top-1 BLEU and Oracle BLEU scores. Crucially, the model achieved significantly lower Self-BLEU scores, indicating a higher degree of diversity in the generated questions. Qualitative analysis shows that the bilevel approach helps prevent hallucinations (e.g., misattributing actions to the wrong subject) by grounding the question in a generated rationale.
Efficiency and Human Evaluation
Despite having only 10% of the parameters, BPD-DSRL outperformed much larger models like LLaMA3-8B and Mistral-7B in zero-shot and few-shot settings. Human evaluations further confirmed the model's superiority, noting lower hallucination rates and higher answerability compared to traditional supervised models.
Conclusion
BPD-DSRL represents a significant step forward in making RL-based question generation both efficient and diverse. By decomposing the policy and adopting a diversity-seeking objective, the framework successfully captures the one-to-many nature of human language. Future work may involve integrating Retrieval-Augmented Generation (RAG) to further enhance diversity and addressing the increased inference latency inherent in the bilevel generation process.
Summary updated