TL;DR
This paper introduces SemPool, a novel graph pooling method that enhances language models (LMs) for question answering (QA) by effectively integrating knowledge graph (KG) information. SemPool addresses the limitations of existing KG-based QA methods, particularly when crucial answer information is missing from the KG, demonstrating improved robustness and interpretability.
What Problem Does This Paper Address?
Question answering (QA) systems aim to understand natural language queries and leverage domain-specific knowledge. While large language models (LMs) excel at understanding semantics, they often struggle with complex reasoning tasks requiring deep, in-domain knowledge. Knowledge Graphs (KGs) offer a structured way to represent factual knowledge, and Graph Neural Networks (GNNs) have been used to integrate KG information with LMs. However, current GNN-based methods often rely heavily on the connectivity of answer nodes within the KG. This makes them vulnerable to "graph perturbations," such as when critical information about candidate answers is missing or removed from the KG. This limitation hinders their robustness and effectiveness in real-world scenarios where KGs might be incomplete or dynamic. Furthermore, the mismatch in representation spaces between graph structures and language sequences can impede effective information exchange.
Methods
SemPool proposes a simple yet effective graph pooling approach to address these challenges. Instead of relying on message-passing mechanisms that propagate information locally, SemPool focuses on aggregating global semantic information from the KG.
Here's a breakdown of the key components:
- KG Initialization: For a given question and candidate answer, a relevant KG subgraph is retrieved.
- Semantic Alignment: Each fact (triplet) within the KG subgraph is encoded into a natural language representation using predefined templates and then embedded using the same pre-trained LM that will be used for QA. This crucial step ensures semantic alignment between the language and graph modalities.
- Pooling: SemPool employs a self-attention pooling layer to aggregate the semantic embeddings of all facts in the subgraph into a single global representation. This pooling mechanism learns to weight the importance of each fact, allowing the model to identify more relevant information.
- KG Grounding: The aggregated KG representation is then fused with the LM's processing. This is achieved by inserting a special
[Graph]token at the beginning of the input sequence, whose embedding is set to the pooled KG representation. This allows the LM's transformer layers to directly learn to fuse graph semantics with language. - Fusion Strategies: SemPool offers two fusion strategies:
- Early Fusion: The pooled KG representation is prepended to the input query.
- Early & Late Fusion: The pooled KG representation is fused at multiple layers of the LM via skip connections, allowing for more nuanced integration of graph information throughout the reasoning process.
Datasets
The paper evaluates SemPool on three standard multiple-choice QA datasets:
- OpenBookQA (OBQA): A 4-way multiple-choice QA dataset requiring reasoning with elementary science knowledge.
- RiddleSense (RiddleQA): A 5-way multiple-choice task testing complex riddle-style commonsense reasoning.
- MedQA-USMLE (MedQA): A 4-way multiple-choice dataset for biomedical and clinical knowledge, derived from USMLE practice sets.
The authors also investigate a challenging setting where critical answer information is removed from the KG to test robustness.
Key Insights and Ideas
- Robustness to Missing Information: The core insight is that SemPool's global pooling approach is inherently more robust to missing answer-specific information in the KG compared to GNNs that rely on local message passing around answer nodes.
- Semantic Alignment is Crucial: Using the same LM for both encoding KG facts and initializing the QA-finetuned LM significantly improves performance. This suggests that aligning the semantic spaces of language and graph is key for effective knowledge transfer.
- Interpretability of Fusion: The multi-layer fusion (late fusion) allows SemPool to extract different types of semantic information at different stages of the LM's reasoning process, offering valuable interpretability into what KG information is being used and when.
- Simplicity and Efficiency: SemPool is a simple pooling mechanism that avoids complex message-passing, making it computationally efficient and easy to integrate into existing LM architectures.
Key Results
- Outperforms GNNs in Adversarial Setting: SemPool outperforms state-of-the-art GNN-based methods by an average of 2.27% accuracy points in the challenging setting where answer information is missing from the KG.
- Competitive Performance: Even when complete answer information is available, SemPool is competitive, often achieving the second-best performance.
- Significant Improvement over LM (w/o KG): SemPool provides a substantial improvement of 4.1% accuracy points over a baseline LM that does not use any KG information.
- LM Agnostic: SemPool demonstrates effectiveness across different LMs, including RoBERTa, AristoRoBERTa, SapBERT, and BioLinkBERT.
- Interpretability: Analysis shows that SemPool can identify relevant KG facts and even highlight irrelevant concepts for incorrect answers, providing insights into its reasoning process.
Limitations and Future Work
- Hyperparameter Tuning: The optimal number of fusion layers (K) is task- and model-specific and requires tuning.
- Subgraph Retrieval: The performance of SemPool is dependent on the quality of the retrieved KG subgraph.
- Future Work: The paper suggests exploring more sophisticated fusion strategies and further investigating the interpretability of SemPool. The authors also hint at combining SemPool with message-passing GNNs for even higher performance, as demonstrated in Table 7.
Summary updated