Ensembling Textual and Structure-Based Models for Knowledge Graph Completion

Ananjan Nandi, Navdeep Kaur, Parag Singla, Mausam

Abstract: We consider two popular approaches to Knowledge Graph Completion (KGC): textual models that rely on textual entity descriptions, and structure-based models that exploit the connectivity structure of the Knowledge Graph (KG). Preliminary experiments show that these approaches have complementary strengths: structure-based models perform exceptionally well when the gold answer is easily reachable from the query head in the KG, while textual models exploit descriptions to give good performance even when the gold answer is not easily reachable. In response, we propose DynaSemble, a novel method for learning query-dependent ensemble weights to combine these approaches by using the distributions of scores assigned by the models in the ensemble to all candidate entities. DynaSemble achieves state-of-the-art results on three standard KGC datasets, with up to 6.8 pt MRR and 8.3 pt Hits@1 gains over the best baseline model for the WN18RR dataset.

Introduction

Knowledge Graph Completion (KGC) involves predicting missing links in a graph, typically represented as a triple $(h, r, ?)$ where the goal is to identify the tail entity $t$. Current research generally falls into two paradigms: structure-based models and textual models.

Structure-based models, such as Graph Neural Networks (GNNs) or path-based algorithms (e.g., NBFNet), excel at exploiting graph topology. They are highly accurate when the target entity is topologically "near" the source node but struggle in sparse regions. Conversely, textual models leverage Pre-trained Language Models (PLMs) like BERT to process entity descriptions. These models remain robust even when graph structure is unhelpful but often lack the precise topological reasoning of their structural counterparts.

Existing methods to combine these paradigms, such as fixed-weight ensembling or simple re-ranking, are often static and fail to account for query-specific nuances. DynaSemble addresses this by dynamically weighting models based on the specific characteristics of each query.

Methodology

DynaSemble is a lightweight, model-agnostic "late fusion" framework that combines the outputs of pre-trained models without requiring expensive retraining of the underlying architectures.

  1. Normalization: To account for different score scales across models, DynaSemble applies max-min normalization to the scores assigned to all candidate entities.
  2. Feature Extraction: For every model, the system calculates the mean and variance of the score distribution across all candidates. A confident model typically exhibits a distribution with low mean and low variance (indicating a clear winner), while an uncertain model produces a noisier distribution.
  3. Dynamic Weighting: These statistical features are processed by a small, 2-layer Multi-Layer Perceptron (MLP) that learns to output a query-dependent weight for each model in the ensemble.
  4. Inference: The final prediction is calculated as a weighted sum of the individual models' scores.

Experimental Evaluation

The framework was evaluated on four standard KGC benchmarks: WN18RR (sparse), FB15k-237 (redundant relations), CoDex-M (general knowledge), and YAGO3-10 (large-scale).

The Reachability Trade-off

A core motivation for DynaSemble is the "reachability" trade-off. Structure-based models dominate when the answer is within a short path (reachable) in the KG, while textual models excel for unreachable nodes. DynaSemble successfully learns to shift its trust between models based on this signal.

Performance comparison on Reachable vs. Unreachable splits across WN18RR, FB15k-237, and CoDex-M.

Key Results and Insights

DynaSemble achieved new state-of-the-art (SOTA) results across multiple benchmarks. On WN18RR, it demonstrated a significant jump of 6.8 points in MRR and 8.3 points in Hits@1 over the previous best individual model (SimKGC).

Main experimental results showing MR, MRR, and Hits@k for DynaSemble and various baselines across four datasets.

Key findings include:

  • Superiority over Static Baselines: DynaSemble consistently outperformed static ensembling and the re-ranking heuristics used in models like KGT5.
  • Model Agnosticism: While primary tests used SimKGC (textual) and NBFNet (structural), the method generalized effectively to other models like RotatE and ComplEx.
  • Soft Ensembling Benefits: Interestingly, DynaSemble's "soft" ensemble often outperformed an "Oracle" that simply picked the single best model, suggesting that aggregating signals from both paradigms provides a synergistic effect.

Conclusion and Future Work

DynaSemble provides a computationally efficient way to harness the complementary strengths of textual and structural KGC models. Future research directions include incorporating learned query embeddings to refine the weighting logic and extending the framework to temporal and multilingual Knowledge Graphs.

Summary updated

Abstract

We consider two popular approaches to Knowledge Graph Completion (KGC): textual models that rely on textual entity descriptions, and structure-based models that exploit the connectivity structure of the Knowledge Graph (KG). Preliminary experiments show that these approaches have complementary strengths: structure-based models perform exceptionally well when the gold answer is easily reachable from the query head in the KG, while textual models exploit descriptions to give good performance even when the gold answer is not easily reachable. In response, we propose DynaSemble, a novel method for learning query-dependent ensemble weights to combine these approaches by using the distributions of scores assigned by the models in the ensemble to all candidate entities. DynaSemble achieves state-of-the-art results on three standard KGC datasets, with up to 6.8 pt MRR and 8.3 pt Hits@1 gains over the best baseline model for the WN18RR dataset.
BibTeX
@inproceedings{nandi2024ensembling,
  author = {Ananjan Nandi and Navdeep Kaur and Parag Singla and Mausam},
  title = {{Ensembling Textual and Structure-Based Models for Knowledge Graph Completion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=rg6SjE2yiu},
}
Figure