TL;DR
This paper tackles the challenge of extracting relevant common sense knowledge from large knowledge graphs for tasks like question answering. They introduce a novel triple-scoring method and two subgraph construction techniques (weighted pathfinding and minimum spanning trees) that significantly improve accuracy by ensuring connectivity and relevance.
What Problem Does This Paper Address?
For AI systems to reason effectively about everyday situations, they need access to common sense knowledge. This knowledge is often stored in large, structured formats called knowledge graphs. However, a major hurdle is efficiently and accurately extracting only the relevant pieces of information from these vast graphs for a specific task, such as answering a question. Simply retrieving all facts is computationally expensive and introduces noise. Existing methods for extracting relevant subgraphs are often simplistic, leading to performance degradation. This paper aims to bridge this gap by developing more sophisticated methods for identifying and constructing these crucial knowledge graph subsets.
Methods
The core of the paper's contribution lies in its approach to identifying and constructing relevant subgraphs.
- Triple Scoring: They frame the problem as a ranking task, scoring individual triples (facts) within the knowledge graph based on their relevance to a given question and candidate answer. They leverage a pre-trained model (from Pan et al., 2021) that excels at identifying explanatory facts, assuming these are also relevant for distinguishing correct answers.
- Subgraph Construction: After scoring triples, they propose two methods to build a connected subgraph:
- Weighted Pathfinding: An extension of prior work, this method iteratively finds shortest paths between concepts in the question and answer, adding nodes and edges.
- Minimum Spanning Tree (MST): This approach constructs a minimum spanning tree over the selected triples, ensuring all nodes are connected. They experiment with both unweighted and weighted MSTs (using the triple scores as weights).
- Concept Identification (Entity Linking): To ensure the extracted subgraphs accurately represent concepts from the question and answer, they compare two entity linking methods:
- Lexical Matching: A heuristic-based approach common in prior work.
- Embedding-based: Using embeddings from a language model (RoBERTa) to find semantically similar concepts.
Datasets
- Open-bookQA: A dataset of science questions. This is considered "in-domain" for the fact scorer used in the paper.
- CommonsenseQA: A dataset targeting general common sense knowledge. This dataset is used to evaluate the transferability of the fact scorer to other domains.
Key Insights and Ideas
- Connectivity Matters: Ensuring the extracted subgraph is connected significantly improves performance, especially when paths are used for model interpretation. Disconnected graphs limit information flow between nodes.
- Ranking is Powerful: A sophisticated ranking model for triples is more effective than simply selecting the highest-ranked ones, suggesting that the nuances of relevance can be captured.
- Embedding-based Entity Linking Can Outperform Lexical: For certain tasks and knowledge graphs, using embeddings to identify concepts can lead to better results than traditional lexical matching, especially when dealing with less common phrasing.
- Domain Transferability: The fact scorer, trained on science questions, shows promising performance on general common sense tasks, hinting at its broader applicability.
Key Results
- On CommonsenseQA, their best method (weighted MST with embedding-based grounding) significantly outperforms the baseline, achieving higher accuracy.
- On Open-bookQA, while the tuned baseline was competitive, their connectivity-ensuring methods (MST and weighted pathfinding) generally outperformed the method that only used the highest-ranked triples.
- Embedding-based entity linking showed a notable increase in score for Open-bookQA compared to lexical linking when using an unweighted MST.
- The weighted MST with embedding grounding resulted in substantially larger subgraphs compared to lexically-linked entities, likely because embedding-based methods identify more specific, less connected concepts.
Limitations and Future Work
- Fact Ranker Influence: The paper acknowledges that the performance of their method is heavily influenced by the quality of the fact ranker. Future work could explore further training or fine-tuning of this ranker, potentially using higher-quality negative samples.
- Domain Adaptation: While the fact ranker showed some domain transferability, further investigation into its performance across diverse domains and potential adaptation strategies is suggested.
- Subgraph Size: The paper notes that subgraph size can be a confounding factor and explores this in the appendix. Future work could investigate more fine-grained control over subgraph size and its impact.
Summary updated