Learning Embeddings from Knowledge Graphs With Numeric Edge Attributes

Sumit Pai, Luca CostabelloIJCAI 2021 · DOI

TL;DR

This paper introduces FocusE, a novel layer for knowledge graph embedding (KGE) models that effectively incorporates numeric attributes associated with edges (triples). By modulating the scoring of triples based on these numeric values, FocusE significantly improves the ability of KGE models to distinguish between high- and low-valued links, leading to better link prediction performance.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are powerful tools for representing structured information, but traditional KGE models often overlook valuable numeric information attached to the relationships (edges) between entities. These numeric attributes can represent crucial details like edge uncertainty, importance, or confidence, which are vital in many real-world applications such as bioinformatics or social networks. Existing KGE models are primarily designed for symbolic data and struggle to leverage these numeric edge attributes, thus limiting their predictive accuracy and ability to capture nuanced relationships. While some work has explored numeric attributes on entities, those focusing on edge attributes are scarce, and existing methods like UKGE require additional external rules and are primarily designed for uncertainty. This paper aims to bridge this gap by proposing a method that directly integrates numeric edge attributes into KGE models without requiring external logic.

Methods

The core contribution is the FocusE layer, designed as a plug-in module that sits between the scoring and loss layers of existing KGE architectures. FocusE works by modulating the output of the traditional scoring function based on the numeric value ($w$) associated with each triple.

Here's a breakdown of the key components:

  • Scoring Function Modulation: FocusE uses a modulating factor ($\alpha$) that balances the influence of the graph's structure (via a hyperparameter $\beta$) and the numeric value of the triple ($w$). This factor is applied to the output of a non-linearity applied to the original KGE model's scoring function ($g(t)$).
  • "Focusing" on High-Value Triples: The modulation is designed such that triples with higher numeric values receive a greater "focus" during training. This is achieved by adjusting the scoring based on $w$, effectively increasing the margin between scores of positive triples and their corruptions when the numeric value is high.
  • Decaying Structural Influence: The method employs a decay mechanism for the structural influence parameter ($\beta$) over training epochs. Initially, $\beta$ is set to 1, giving equal importance to structural information and numeric attributes. As training progresses, $\beta$ decays to 0, making the model increasingly reliant on the numeric attributes. This allows the model to first learn general structural patterns and then refine its understanding by focusing on the numeric edge information.
  • Loss Function Integration: The modulated scores from the FocusE layer are then used in a modified loss function, which is a numerically stable version of the negative log-likelihood of normalized softmax scores.

Datasets

The authors evaluated their approach on four datasets:

  • CN15K: A subset of ConceptNet, with numeric values representing uncertainty.
  • NL27K: A subset of the NELL dataset, with numeric values representing link uncertainty.
  • PPI5K: A knowledge graph of protein-protein interactions, with numeric values representing confidence.
  • O*NET20K: A newly introduced dataset (subset of O*NET) containing job descriptions and skills, where numeric values represent the importance of a link. This dataset was specifically curated to mock up single-target link prediction tasks.

All datasets included triple-specific numeric values, which were normalized to the [0, 1] range.

Key Insights and Ideas

  • Numeric Edge Attributes are Crucial: The paper strongly argues that numeric edge attributes are not just noise but contain vital information that traditional KGE models miss, significantly impacting predictive power.
  • "Focusing" as a Training Strategy: The core idea of modulating scores based on numeric values to "focus" training on more informative triples is a novel and effective way to leverage this data.
  • Decay Mechanism for Gradual Learning: The gradual decay of structural influence allows the model to first learn general KG patterns and then specialize in using numeric edge information, providing a more robust learning process.
  • Generalizability of Numeric Semantics: FocusE works regardless of the specific semantic meaning of the numeric values (uncertainty, importance, confidence), highlighting its broad applicability.

Key Results

  • Improved Link Prediction: FocusE consistently outperformed traditional numeric-unaware baselines (TransE, DistMult, ComplEx) across all datasets in predicting high-valued links.
  • Superior Discrimination: FocusE demonstrated a significantly better ability to discriminate between high-valued and low-valued triples compared to baselines and the closest competitor, UKGE. This is evidenced by larger differences in Mean Reciprocal Rank (MRR) between top-10% and bottom-10% valued triples.
  • Outperforming UKGE: FocusE achieved better predictive power than UKGE, particularly in discriminating high- and low-valued links, and crucially, without requiring additional external logical rules.
  • Specific Performance Gains: For instance, on CN15K, FocusE improved MRR by 15 base points over baselines. On PPI5K, it showed up to a 30-point MRR improvement and a 411% increase in discriminating high- from low-valued links compared to UKGE.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that UKGE requires additional external logical rules, which are not available for all datasets (e.g., O*NET20K), limiting its evaluation on certain benchmarks.
  • The current implementation focuses on a single numeric attribute per triple.
  • Future Work:
    • Investigate the capability of predicting numeric values associated with unseen triples.
  • Extend the approach to support multiple numeric attributes associated with the same triple.

Summary updated

BibTeX
@inproceedings{pai2021learning,
  author = {Sumit Pai and Luca Costabello},
  title = {{Learning Embeddings from Knowledge Graphs With Numeric Edge Attributes}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/395},
}
Figure