Abstract
Vidhisha Balachandran, Bhuwan Dhingra, Haitian Sun, Michael Collins, William Cohen. Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures.
2021.
Introduction
While Knowledge Bases (KBs) like Wikidata contain a wealth of structured information, leveraging them to improve performance on real-world Question Answering (QA) tasks remains a significant challenge. Most research in Knowledge Base Question Answering (KBQA) relies on "toy" datasets where questions are specifically designed to be answerable by a KB. In contrast, natural questions—the kind users actually type into search engines—are often messy, ambiguous, and require a combination of textual evidence and structured facts.
This paper investigates whether injecting KB "paths" (sequences of related entities and relations) into a standard text-based QA system can improve performance on the Natural Questions (NQ) benchmark. The authors specifically explore the gap between potential gains from "perfect" KB facts and the reality of current retrieval methods.
Methods
The researchers propose a model-agnostic approach to KB integration consisting of three main steps:
- KB-to-Text: Structured triples from Wikidata (e.g., Imran Khan -> officeholder -> Prime Minister) are converted into simple natural language sentences.
- Context Augmentation: These "fact sentences" are simply appended to the textual passage provided to a BERT-based QA model, allowing it to attend to both sources simultaneously.
- Retrieval Strategies: To select relevant facts from the massive KB, the authors compare several Personalized PageRank (PPR) variants:
- Traditional PPR: A random walk starting from entities in the question.
- Question Informed (QI) PPR: Weights the walk based on similarity between question text and KB relations.
- Weakly Supervised (WS) PPR: A novel approach where a BERT classifier predicts relevant KB relations to guide the PPR walk.
To establish an upper bound, an Oracle retriever was created to provide the shortest KB path between the question entity and the correct answer.
Experimental Setup
The study utilizes the Natural Questions (NQ) dataset and introduces a new subset called Factual Questions (FQ). FQ consists of approximately 7,000 NQ examples where the answer is a known entity and a short path (1-3 hops) exists in Wikidata. This subset allows for a controlled evaluation of KB integration without the interference of non-factual queries.
Key Results
The Potential of Background Knowledge
The "Oracle" experiments reveal the significant potential of KB integration. When provided with the correct KB path, performance on the FQ subset improved dramatically. Short-answer F1 scores increased by 6.7 points overall. Notably, the gains were most pronounced for complex "multi-hop" questions requiring 2 or 3 steps of reasoning, where the KB acts as a logical shortcut for information that might be fragmented across text.
The Retrieval Bottleneck
Despite the potential shown by the Oracle, automated retrieval remains a major hurdle. Even the advanced WS-PPR method only successfully retrieved the correct answer entity about 40% of the time. While WS-PPR significantly outperformed traditional PPR and BM25 baselines in terms of recall, the noise introduced by irrelevant facts often offset the benefits of the correct ones on the full NQ dataset.
Discussion and Conclusion
The findings suggest that KBs are most helpful for multi-hop reasoning where textual evidence might be fragmented. However, the primary limitation to practical KB integration is the retrieval gap. Current heuristics and PageRank-based methods struggle to reliably extract the necessary facts for open-ended questions.
The authors conclude that simply appending facts as text is an effective integration strategy, but future progress depends on developing more robust, supervised "dense retrieval" models for KBs to overcome the limitations of current graph-walk methods.
Summary updated