TL;DR
This paper investigates whether knowledge graph embeddings (KGEs) trained solely for link prediction truly capture the rich structure of knowledge graphs. They introduce new graph-structure prediction tasks and find that multi-task training, often excluding link prediction, significantly improves both intrinsic graph understanding and downstream task performance, especially in low-data scenarios.
What Problem Does This Paper Address?
Knowledge graph embeddings (KGEs) are crucial for representing entities and relations in knowledge graphs (KGs), enabling reasoning and powering various downstream applications. However, the vast majority of KGE research and development has focused almost exclusively on the link prediction task (e.g., predicting a missing entity in a triple like "Austin, capitalOf, ?"). This narrow focus raises a critical question: do KGEs optimized for link prediction actually capture a broader understanding of the KG's structure, and are they truly the most effective representations for diverse downstream tasks? The authors argue that this assumption is largely unexplored and potentially flawed, leading to suboptimal KGE representations.
Methods
The authors propose a two-pronged approach to evaluate KGEs beyond link prediction:
- Intrinsic Evaluation with Graph-Structure Prediction Tasks: They introduce a novel set of tasks designed to assess how well KGE models capture different aspects of KG structure. These tasks include:
- Relation Prediction (REL): Predicting the relation between two entities.
- Domain/Range Prediction (DOM): Predicting the domain or range of a relation.
- Entity Neighborhood Prediction (NBE): Predicting entities related to a given entity.
-
Relation Neighborhood Prediction (NBR): Predicting relations an entity is involved in. These tasks are designed to be simpler than link prediction and have a single prediction target, facilitating efficient training and evaluation. They also introduce a Multi-Task Ranking (MTR) protocol to aggregate performance across these new tasks.
-
Extrinsic Evaluation on Downstream Tasks: They evaluate pre-trained KGEs on 35 diverse entity classification and regression tasks. This allows them to assess the practical utility of the learned representations.
A key methodological contribution is Multi-Task Training (MTT), where KGE models are trained simultaneously on link prediction and the newly proposed graph-structure prediction tasks. They also explore different combinations of these tasks for pre-training.
Datasets
The study utilizes several well-established benchmark datasets for KGE research:
- FB15K-237: A popular dataset for link prediction and downstream tasks.
- WNRR: Another challenging dataset for link prediction.
- YAGO3-10: A larger dataset for KGE tasks.
- WIKIDATA5M: A very large dataset.
The authors also collected/created data for 35 downstream tasks across these datasets for entity classification and regression.
Key Insights and Ideas
- Link Prediction is Not a Panacea: Strong performance on link prediction does not necessarily translate to good performance on graph-structure prediction tasks or downstream applications. This challenges the long-held assumption in the field.
- KGEs Can Be "Blind" to Structure: Standard KGE models, trained only on link prediction, often perform poorly on tasks that require understanding broader graph structures (like relation prediction or neighborhood prediction). This suggests they might be overfitting to the specific patterns needed for link prediction.
- Multi-Task Training is Key: Training KGEs on a combination of tasks, especially those that probe graph structure, significantly improves their ability to capture KG properties and their utility in downstream tasks.
- Excluding Link Prediction Can Help: Surprisingly, excluding the link prediction task from the pre-training objective sometimes leads to better downstream performance, suggesting that LP can even be detrimental in some multi-task settings.
- Data Efficiency Matters: MTT models show improved data efficiency, maintaining better performance than standard (STD) models in few-shot or downsampled training scenarios, particularly for classification tasks.
Key Results
- MTT models consistently outperformed STD models on the new graph-structure prediction tasks, often by a significant margin (2x to 10x).
- For downstream tasks, MTT models generally achieved better performance than STD models, especially when data was scarce. For example, in few-shot classification, MTT models showed a clear advantage.
- The best performing models for link prediction were often not the best for downstream tasks. For instance, TransE (a simpler model) sometimes outperformed more complex models like ComplEx or RotatE on downstream tasks, despite weaker link prediction scores.
- The optimal choice of pre-training tasks is highly dependent on the specific KGE model, the KG dataset, and the downstream task.
Limitations and Future Work
- Limited Downstream Task Variety: The study focused on entity-level classification and regression. The authors suggest exploring the impact of pre-training on more complex downstream applications like recommender systems or question answering.
- No Concrete Solution for Generalizability: While MTT shows promise, the paper doesn't provide a definitive "recipe" for selecting pre-training tasks to maximize generalizability. This remains a challenging problem due to the complexity and diversity of KG structures.
- Exploration of Other Tasks: The authors suggest exploring other self-supervised tasks, such as predicting n-hop neighborhoods or objectives that more closely resemble downstream tasks.
- Combining with Supervised Training: Future work could explore combining these self-supervised objectives with supervised training objectives.
Summary updated