OKGIT: Open Knowledge Graph Link Prediction with Implicit Types

Chandrahas, Partha Pratim TalukdarACL 2021

TL;DR

This paper introduces OKGIT, a novel method for predicting missing links in Open Knowledge Graphs (OpenKGs). By leveraging the implicit type information found in pre-trained language models like BERT, OKGIT significantly improves the accuracy and type compatibility of link predictions, outperforming existing methods.

What Problem Does This Paper Address?

Open Knowledge Graphs (OpenKGs) are valuable resources for various downstream NLP tasks, but they suffer from sparsity and a lack of explicit type information for entities. This makes it challenging to predict new facts (link prediction) accurately. Existing methods often predict entity pairs that are factually correct but semantically incompatible (e.g., predicting a person as a location). This paper addresses the critical problem of improving the type compatibility of link predictions in OpenKGs, ensuring that predicted relationships make semantic sense.

Methods

The core of OKGIT lies in its novel approach to integrate type information into the link prediction process. It builds upon existing OpenKG embedding methods like CaRE, which use a standard triple scoring function. OKGIT enhances this by:

  • Leveraging BERT for Type Information: It uses a pre-trained BERT model to predict a "masked" tail entity's embedding. This BERT embedding, while not always factually correct, is hypothesized to contain rich, implicit type information.
  • Type Projectors: To extract this type information effectively, OKGIT employs "type projectors" – learned matrices that map BERT's tail entity embeddings and the standard embedding vectors into a common type space.
  • Type Compatibility Score: A new score is introduced that measures the similarity (specifically, the negative Euclidean distance) between the type vector of the predicted tail entity (from BERT) and the type vector of the actual tail entity (from the standard embedding).
  • Combined Score: The final prediction score for a triple is a weighted combination of the original triple score (measuring factual correctness) and the new type compatibility score.
  • Type Regularization: An additional loss term is introduced to encourage type vectors of correct triples to be closer in the type space, further reinforcing type compatibility.

Datasets

The paper evaluates OKGIT on two subsets of English OpenKGs derived from the ReVerb dataset:

  • ReVerb20K and ReVerb45K: These are standard datasets for OpenKG link prediction.
  • ReVerb20KF and ReVerb45KF: Filtered subsets containing only single-token noun phrases (NPs) to specifically test BERT's capabilities, as BERT is primarily designed for single tokens.

Key Insights and Ideas

  • Implicit Type Information in BERT: The paper strongly suggests that pre-trained language models like BERT, even without fine-tuning, contain valuable implicit type information about entities that can be exploited for downstream tasks.
  • Type Compatibility is Crucial: Simply predicting factually correct links isn't enough. For OpenKGs to be truly useful, the predicted links must also be semantically plausible, meaning the entities involved should be of compatible types.
  • Combining Factual and Semantic Scores: The success of OKGIT highlights the benefit of combining a traditional link prediction score (factual correctness) with a novel type compatibility score (semantic plausibility).
  • Type Projectors are Effective: The learned type projectors are shown to be effective in extracting meaningful type-specific representations from entity embeddings, separating entities into distinct type clusters.

Key Results

  • State-of-the-Art Performance: OKGIT consistently outperforms baseline methods (CaRE, ConvE, BERT) across all datasets in terms of Mean Reciprocal Rank (MRR) and Hits@k metrics. For example, on ReVerb20K, OKGIT achieves an MRR of 35.9%, compared to CaRE's 30.6%.
  • Improved Type Compatibility: Qualitative evaluations and quantitative F1-scores (using an external entity typing model) show that OKGIT generates significantly more type-compatible predictions than CaRE.
  • Ablation Studies: Both the type compatibility score and type regularization term are shown to be important, with the combined model achieving the best performance.

Limitations and Future Work

  • Dependency on Pre-trained Models: The effectiveness of OKGIT relies on the quality and type information present in the pre-trained language model (e.g., BERT).
  • External Entity Typing Model Limitations: The evaluation of type compatibility in Section 6.2 relies on an external entity typing model (UFET), and its coverage and quality can limit the evaluation.
  • Single-Token Limitation for BERT: While OKGIT itself can handle multi-token NPs, the BERT module used for type prediction in the original formulation has a limitation to single tokens. The paper notes that this limitation is not inherent to OKGIT's overall architecture.
  • Language Dependency: The model's effectiveness might vary across languages, and it has only been tested on English datasets.

Future work could explore using other pre-trained language models, investigating multi-lingual OpenKG link prediction, and developing more robust methods for evaluating type compatibility without relying on external typing models.

Summary updated

BibTeX
@inproceedings{chandrahas2021okgit,
  author = {Chandrahas and Partha Pratim Talukdar},
  title = {{OKGIT: Open Knowledge Graph Link Prediction with Implicit Types}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure