KCS: Diversify Multi-hop Question Generation with Knowledge Composition Sampling

Yangfan Wang, Jie Liu, Chen Tang, Lian Yan, Jingchi JiangEMNLP 2025

TL;DR

This paper introduces Knowledge Composition Sampling (KCS), a novel framework for generating diverse multi-hop questions. KCS addresses data sparsity in question answering by intelligently sampling varied combinations of knowledge snippets from a given context, leading to more robust and generalizable question generation models.

What Problem Does This Paper Address?

Multi-hop question answering (MHQA) requires models to synthesize information from multiple sources to answer complex questions. A significant challenge in this field is data sparsity, which can lead language models to learn spurious patterns and hinder their generalization. While previous work has focused on diversifying question generation through content planning and expression variation, these methods often overlook the crucial step of integrating relevant external knowledge, such as specific sentences from documents. This paper tackles the problem of generating diverse and high-quality multi-hop questions by focusing on diversifying the knowledge compositions used to construct these questions.

Methods

The core of KCS is its three-component framework:

  1. Knowledge Composition Selection: This component uses a hierarchical neural network (encoder-decoder architecture) to predict "question-worthy" sentences from a given context. It frames this as a sentence-level sequence prediction task, employing a probabilistic contrastive loss to learn potential knowledge coherence. This loss function aims to maximize the mutual information between the current and next predicted knowledge pieces while minimizing it with other contextually irrelevant pieces.
  2. Diversifying Knowledge Composition: To ensure diversity, KCS employs a stochastic decoding strategy, specifically nucleus sampling. This method truncates the less probable parts of the prediction distribution and samples from a dynamic nucleus of sentences that capture the majority of the probability mass. This allows for the selection of varied knowledge compositions even when the underlying probability distribution might be similar.
  3. Multi-hop Question Generation: Once diverse knowledge compositions are sampled, a standard MHQG model (fine-tuned on a vanilla model) is used to generate a question conditioned on the selected knowledge and the given answer.

Datasets

The paper evaluates its method on two widely-used MHQA datasets:

  • HotpotQA: A benchmark dataset for multi-hop question answering.
  • 2WikiMultihopQA: Another benchmark dataset for multi-hop question answering.

For knowledge composition selection, the development sets were used as test sets, and 500 samples from the training set were used for development. Samples with "yes" or "no" answers were excluded. For data augmentation experiments, 5000 samples from the training set were used for augmentation, and a filtered test set of 200 samples was created.

Key Insights and Ideas

  • Knowledge Composition is Key to Diversity: The paper argues that the diversity of generated multi-hop questions is heavily influenced by the diversity of the underlying knowledge compositions used. Even small variations in selected knowledge snippets can lead to significantly different questions.
  • Probabilistic Contrastive Loss for Coherence: The use of a probabilistic contrastive loss is crucial for learning the "potential knowledge coherence" within a selected set of sentences, preventing the generation of nonsensical or irrelevant questions.
  • Stochastic Decoding Balances Accuracy and Diversity: Nucleus sampling effectively balances the need for accurate knowledge selection with the goal of generating diverse compositions, avoiding the pitfalls of purely greedy or overly random approaches.
  • Unstructured Text for Coherence: KCS leverages unstructured text to discern knowledge coherence, offering greater flexibility and scalability compared to methods relying on pre-defined knowledge graphs.

Key Results

  • Knowledge Composition Selection: KCS improves the overall accuracy of knowledge composition selection by 3.9% compared to competitive baselines, specifically outperforming the CLS (classification) baseline by about 1.5-3% on HotpotQA and 2.1-7.2% on 2WikiMultihopQA.
  • Data Augmentation: When used for data augmentation, KCS consistently improves downstream MHQA performance on both HotpotQA and 2WikiMultihopQA.
  • LLM Evaluation: LLM evaluation scores for KCS-diversified compositions achieved 91.55% and 93.32% of the ground truth scores on HotpotQA and 2WikiMultihopQA, respectively, demonstrating logical coherence.
  • Diversity vs. Consistency: KCS achieves high diversity in generated questions, with consistency being a factor that can be further improved by advanced MHQG models.

Limitations and Future Work

  • Optimization Potential: The authors suggest exploring different mutual information calculation methods, model architectures, and leveraging more high-quality annotated data to further optimize KCS.
  • Domain-Specific Data: Investigating KCS's performance on domain-specific data could help mitigate data sparsity challenges in specialized fields.
  • Data Value: Further research is needed to understand what constitutes "valuable" diverse data for downstream task improvement.
  • MHQG Model Dependency: The consistency of generated questions is currently limited by the vanilla MHQG model used. Future work could integrate more advanced MHQG methods to improve this aspect.

Summary updated

BibTeX
@inproceedings{wang2025kcs,
  author = {Yangfan Wang and Jie Liu and Chen Tang and Lian Yan and Jingchi Jiang},
  title = {{KCS: Diversify Multi-hop Question Generation with Knowledge Composition Sampling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure