Knowledge Transfer between Structured and Unstructured Sources for Complex Question Answering

Lingbo Mo, Zhen Wang, Jie Zhao, Huan SunNAACL 2022

TL;DR

This paper introduces SIMULTQA, a unified framework for complex question answering (CQA) that bridges the gap between structured knowledge bases (KBs) and unstructured text corpora. They demonstrate that knowledge learned from one source can significantly boost performance on the other, especially in low-resource settings, opening avenues for more efficient CQA system development and dataset annotation.

What Problem Does This Paper Address?

Complex Question Answering (CQA) systems aim to answer questions that require reasoning over multiple pieces of information. Traditionally, CQA systems have been developed separately for two main types of knowledge sources: structured knowledge bases (like Freebase) and unstructured text corpora (like Wikipedia). While both approaches have achieved impressive results, the potential for knowledge transfer between these distinct systems and sources has been largely unexplored. This paper tackles the challenge of understanding and enabling this cross-source knowledge transfer, asking: can a system trained on text data help answer questions using a knowledge base, and vice versa? This is particularly relevant for low-resource scenarios where obtaining labeled data for one source might be expensive.

Methods

The core contribution is the SIMULTQA framework, designed to unify and enable knowledge transfer between KB-based QA (KBQA) and text-based QA (TextQA). SIMULTQA operates in two stages:

  1. Reasoning Path Construction: This stage iteratively searches for evidence from either the text corpus or the KB. For text, it uses TF-IDF to retrieve relevant documents. For KBs, it generates query graphs. The key innovation here is the shared architecture and parameters (BERT encoder and RNN module) used for processing both text and KB evidence, allowing them to learn a common representation.
  2. Reranking and Answer Generation: Candidate reasoning paths generated in the first stage are reranked to select the most promising one for answering the question. The framework then uses this selected path to generate the final answer.

The authors emphasize that SIMULTQA can be trained using supervisions from both KB and text sources simultaneously, facilitating the study of knowledge transfer through pre-training and fine-tuning.

Datasets

The paper uses two prominent CQA datasets:

  • Complex WebQuestions (CWQ): A large-scale KB dataset.
  • HotpotQA: A text-based dataset.

These datasets are chosen for their representative nature and the diversity of reasoning types they encompass. The authors focus on open-domain QA settings for both.

Key Insights and Ideas

  • Cross-Source Transfer is Real and Beneficial: The most significant finding is that pre-training SIMULTQA on one source (e.g., text) and fine-tuning on another (e.g., KB) consistently improves performance, especially when the fine-tuning dataset is small. This suggests that the underlying reasoning mechanisms learned are transferable.
  • Low-Resource Boost: The transfer effect is most pronounced in low-resource settings, highlighting the practical utility of this approach for scenarios with limited labeled data.
  • Shared Semantic Space: Fine-grained analysis suggests that SIMULTQA learns a shared semantic space for reasoning, enabling transfer even when surface forms of evidence or reasoning paths differ significantly between sources.
  • Reasoning Type Transfer: Specific reasoning types, like composition and conjunction, show the most significant performance gains from transfer, indicating that the model learns to perform these common reasoning patterns across different knowledge sources.
  • Question Similarity is Not the Key: Interestingly, the degree of similarity between testing questions and training questions did not strongly correlate with the performance gain from transfer, suggesting that the transfer operates at a more abstract, semantic level.

Key Results

  • Text to KB Transfer: Pre-training on HotpotQA (text) and fine-tuning on CWQ (KB) resulted in an approximate 8 F1 score improvement in a zero-shot setting.
  • KB to Text Transfer: Pre-training on CWQ (KB) and fine-tuning on HotpotQA (text) yielded about a 12.5 F1 improvement in a zero-shot setting.
  • Few-Shot Performance: Figures 4 and 5 demonstrate clear performance gains in few-shot scenarios (e.g., with <100 samples) when transfer learning is applied.
  • Reasoning Type Impact: Composition and conjunction reasoning types showed the largest benefits from transfer learning.
  • Hybrid Evaluation: While not the primary focus, preliminary results in Table 4 suggest that incorporating text paths can improve KB-based QA performance (1.8 F1 gain on CWQ), but KB paths had less impact on text-based QA, possibly due to dataset incompatibilities.

Limitations and Future Work

  • Dataset Incompatibility: The authors acknowledge that certain types of knowledge and reasoning present in one dataset (e.g., constraints or arithmetic operations in CWQ, semantic knowledge not in KB for HotpotQA) are not well-represented in the other, limiting the extent of transfer.
  • Focus on Retriever: The experiments primarily focused on the retriever module for transfer learning. Future work could explore transfer in other modules.
  • Future Directions:
    • Efficient Dataset Annotation: Leveraging pre-trained SIMULTQA to generate high-quality reasoning paths for new datasets to reduce annotation costs.
  • More Diverse Reasoning Types: Encouraging future CQA datasets to include more complex reasoning types beyond composition and conjunction.
  • Universal Reasoning Module: Developing a modular, universal reasoning process that can be injected into future QA systems, leveraging shared underlying reasoning capabilities.

Summary updated

BibTeX
@inproceedings{mo2022knowledge,
  author = {Lingbo Mo and Zhen Wang and Jie Zhao and Huan Sun},
  title = {{Knowledge Transfer between Structured and Unstructured Sources for Complex Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure