TL;DR
This paper introduces ENGRAM, a novel approach for inductive reasoning on knowledge graphs (KGs) with very few examples (few-shot). ENGRAM leverages task-aware language models to bridge the gap between unstructured text and structured KG data, enabling more efficient and accurate reasoning, especially for newly emerging entities.
What Problem Does This Paper Address?
Knowledge graphs (KGs) are constantly growing, with new entities and relationships appearing frequently. However, these new additions often come with very limited associated information (few-shot). Traditional KG reasoning methods, like those based on knowledge graph embeddings (KGE), struggle with this scarcity because they are transductive, meaning they can't easily generalize to unseen entities. Graph Neural Networks (GNNs) offer inductive capabilities but can produce ambiguous features from sparse data. While language models (LMs) have shown promise, they often require manual prompt engineering and suffer from inefficient classification-based inference, where they must check every possible candidate triple. This paper aims to overcome these limitations by enabling LMs to perform inductive reasoning on few-shot KGs more effectively and efficiently.
Methods
ENGRAM introduces several key innovations:
- Task-Aware Language Model with In-Context Learning: Instead of relying on manual prompt templates, ENGRAM uses a task-guided prompting method. It provides the LM with task-specific examples (few associated triples) as in-context prompts. This allows the LM to learn directly from the data and understand structured KG tasks without explicit template design.
- Generative Strategy for Link Prediction: ENGRAM reframes link prediction from a classification problem to a generative one. This means the LM directly produces candidate entities, significantly reducing computational overhead by avoiding the need to iterate through all possible entities.
- Contextual Representation Pool (CRP): To efficiently score candidate entities, ENGRAM pre-computes and stores textual representations of observed entities in a CRP. This allows for simultaneous scoring of all candidates, moving from linear complexity to constant complexity for this step.
- Distant Neighborhood Awareness: To combat the sparsity of structural information for few-shot entities, ENGRAM incorporates features from both direct and second-order (distant) neighbors. A gating mechanism adaptively fuses these features, creating a richer representation.
- Knowledge Distribution Alignment: Recognizing that LMs are trained on general knowledge while KGs are domain-specific, ENGRAM uses a strategy of fine-tuning prefix vectors for each layer of the LM while keeping core parameters frozen. This helps align the LM's knowledge with the KG's domain.
- Joint Training: The model combines textual and structural information by jointly training the LM and GNN components, using a scoring function that balances contributions from both.
Datasets
The paper evaluates ENGRAM on two standard few-shot link prediction benchmark datasets:
- FB15K-237: Contains 5,000 emerging entities with 88,178 associated triples.
- NELL-995: Includes 3,000 emerging entities with 31,873 associated triples.
Textual descriptions for entities and relations were sourced from KG-BERT.
Key Insights and Ideas
- In-Context Learning is Key: The paper strongly suggests that by providing relevant examples directly within the prompt, LMs can effectively learn to perform structured KG tasks, bypassing the need for complex manual prompt engineering.
- Generative Approach Outperforms Classification: Shifting from classifying triples to generating entities dramatically improves efficiency by avoiding exhaustive candidate checks.
- Distant Neighbors Matter: Incorporating information from entities beyond immediate neighbors significantly enhances the structural understanding of sparse few-shot entities.
- Knowledge Alignment is Crucial for LMs: The performance of large LMs like Llama3 can be hindered by a mismatch between their general knowledge and the specific domain of the KG, highlighting the importance of methods like knowledge distribution alignment.
Key Results
ENGRAM achieves state-of-the-art performance on both FB15K-237 and NELL-995 datasets in few-shot scenarios.
- Performance: It significantly outperforms current GNN-based and LM-based methods.
- Efficiency:
- The number of tunable parameters is approximately 1% of previous LM-based methods.
- Inference time is reduced by 1/10 compared to previous methods.
- Computational complexity during inference is reduced from O(|E|) to O(1) for candidate scoring.
Limitations and Future Work
- GPU Memory Consumption: A significant limitation is the substantial GPU memory required for the Contextual Representation Pool, especially when dealing with large-scale KGs or larger language models.
- Future Work: The authors suggest developing more flexible and scalable feature scheduling mechanisms to mitigate memory usage during inference. They also highlight the need for further investigation into the knowledge distribution bias observed with large LMs.
Summary updated