Introduction
Inductive Link Prediction (ILP) is a critical task in Knowledge Graph (KG) completion, focusing on predicting new facts involving entities not seen during the training phase. While progress in this field has been rapid, recent research suggests that the standard evaluation protocols used to benchmark these models are fundamentally flawed. This paper presents a critical reevaluation of current ILP evaluation methods, demonstrating how they can lead to misleading performance metrics and favor models that rely on simple heuristics rather than deep relational reasoning.
The Flaw in Random Sampling
The current "gold standard" for evaluating ILP models involves ranking a correct entity against a small set of 50 randomly sampled negative entities. The authors argue that this protocol is "heavily flawed" because it primarily measures a model's ability to perform broad category or "type" matching rather than specific link prediction.
As shown in Figure 1, relations like currency or capital have specific entity types as valid targets (e.g., currencies or cities). If a model learns that the answer to a currency query must be a currency, it can easily rank the correct answer (e.g., euro) above random negatives like Paris or iPhone without understanding the specific relationship to the subject entity. This "Type Trap" allows simple models to appear sophisticated by merely performing type-checking.
Proposed Evaluation Framework
To address these shortcomings, the authors introduce three key components:
1. Rule-Based Baseline
A simple rule-based system was developed to serve as a benchmark for how much performance can be attributed to basic type-matching. This system learns logical rules to determine valid entity types for specific relations.
Figure 5 illustrates examples of such rules ($r_1, r_2, r_3$) and their associated confidence scores. Surprisingly, this baseline outperformed many state-of-the-art (SOTA) models under the old random sampling protocol, achieving an average Hits@10 of 0.938 on FB15k-237. This proved the old benchmark was not measuring "predictive accuracy" in any meaningful way.
2. Non-Sampling Protocol
The authors propose moving away from sampling and instead ranking the correct entity against every possible entity in the dataset. This is the standard in transductive link prediction and provides a much more rigorous test of a model's predictive power.
3. Type-Matched Negatives (TMN)
Recognizing that ranking against thousands of entities can be computationally prohibitive for some models, the authors introduce the TMN protocol. This involves ranking against 50 "hard" negatives that are all of the correct type for the query, forcing the model to distinguish between plausible candidates based on graph structure.
Experimental Results and Insights
The study re-evaluated 13 SOTA methods across common benchmarks: FB15k-237, WN18RR, and NELL-995. The results revealed a significant "Leaderboard Shuffle":
- Performance Collapse: Models like NodePiece, which performed exceptionally well under random sampling (Hits@10 of 0.926), saw their performance drop by nearly 50% (to 0.450) under the non-sampling protocol. This suggests a heavy reliance on simple 1-hop type information.
- Robust Models: Models such as NBFNet, AStarNet, and RED-GNN emerged as more reliable, maintaining strong performance across the more rigorous protocols.
- Symbolic Resilience: AnyBURL, a symbolic rule-based approach, proved to be highly competitive and robust, offering a balance of efficiency and explainability.
Conclusion
The findings of this reevaluation serve as a call to action for the KG research community. The authors strongly advocate for the abandonment of random sampling in favor of non-sampling or TMN protocols. By adopting these more rigorous standards, researchers can ensure that progress in inductive link prediction reflects genuine improvements in relational reasoning rather than simple exploitation of dataset biases.
Summary updated