Unsupervised Embedding Enhancements of Knowledge Graphs using Textual Associations

Neil Veira, Brian Keng, Kanchana Padmanabhan, Andreas G. VenerisIJCAI 2019 · DOI

TL;DR

This paper introduces two unsupervised methods to improve knowledge graph embeddings by incorporating information from text. By augmenting entity embeddings with associated word embeddings, the authors demonstrate significant improvements in link prediction accuracy across various existing knowledge graph embedding models, making them more robust and capable of inferring new facts.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are powerful for representing structured data and inferring new facts, but they are often incomplete. Traditional KG embedding methods rely solely on the existing facts, limiting their predictive power. While incorporating external information like text is a promising avenue, existing methods often require labeled data or modify the core KG embedding optimization objective. This paper addresses the challenge of leveraging readily available textual data in an unsupervised manner to enhance KG embeddings without altering their fundamental learning process.

Methods

The paper proposes two distinct methods for incorporating textual information, depending on how text is associated with entities:

  • Entity Descriptions (WWV & PE-WWV): When each entity has a descriptive text (e.g., Wikipedia entry), the authors propose augmenting entity embeddings with weighted word vectors from these descriptions.
  • WWV (Weighted Word Vectors): This method learns a relation-specific weight matrix (B) that determines how much each word in an entity's description contributes to its embedding for a particular relation. This allows entities to have different representations depending on the relation being considered.
  • PE-WWV (Parameter-Efficient Weighted Word Vectors): To address the large number of parameters in WWV, PE-WWV introduces a more efficient scheme by learning a relation-specific vector (P) that influences word weights, reducing the parameter count significantly while maintaining comparable performance.
  • Unstructured Corpus (FeatureSum): When entities are mentioned in a general text corpus without direct descriptions, the authors leverage word2vec to learn embeddings for entity names.
  • FeatureSum: This model augments the original entity embedding by adding a transformed word2vec embedding of the entity's name. A learnable matrix (M) maps word embeddings to the entity embedding space, allowing the model to capture textual features relevant for KG inference.

All proposed methods are designed to be integrated seamlessly with existing KG embedding models by simply replacing the entity embeddings in their scoring functions.

Datasets

The experiments were conducted on two widely used knowledge graph datasets:

  • Wordnet (WN18): A subset of the WordNet lexical database, containing 18 relations, 40,943 entities, and 156,442 facts. Entity descriptions were derived from WordNet definitions.
  • Freebase (FB15k): A large, collaboratively created graph of real-world knowledge, consisting of 1,345 relations, 14,951 entities, and 592,213 facts. Entity descriptions were sourced from Wikipedia summaries.

For the unstructured corpus experiments, the authors used the Google News dataset and pre-trained word2vec vectors.

Key Insights and Ideas

  • Textual context is crucial: The paper strongly validates the intuition that textual associations can significantly enhance KG embeddings, even when the text is not explicitly structured or labeled for relation extraction.
  • Relation-specific word weighting is powerful: The WWV model's ability to assign different importance to words based on the relation being considered is a key insight. This allows for more nuanced entity representations that capture relational semantics.
  • Parameter efficiency matters: The PE-WWV model demonstrates that achieving high performance doesn't always require a massive increase in parameters, offering a practical advantage for scalability.
  • Unsupervised learning is effective: The success of these methods highlights the power of unsupervised learning for extracting valuable information from text to complement structured KGs.
  • Data quality impacts results: The performance gains from textual augmentation are shown to be dependent on the richness and relevance of the textual descriptions, with Freebase (using Wikipedia summaries) showing more pronounced improvements than Wordnet (using definitions).

Key Results

  • Consistent improvements: Both WWV and PE-WWV consistently outperformed the baseline (no text) and a simpler word vector approach (WV-desc) on link prediction tasks across multiple KG embedding models (SE, TRANSE, TRANSR, RESCAL, DISTMULT, HOLE) on both WN18 and FB15k datasets.
  • PE-WWV rivals WWV: Surprisingly, the parameter-efficient PE-WWV model often matched or even surpassed the performance of the more complex WWV model, suggesting that the optimization process might be more sensitive to how weights are structured than to the sheer number of parameters.
  • FeatureSum outperforms baselines: For the unstructured corpus scenario, the FeatureSum model generally outperformed other text-based methods (WV-names, WV-names-init), demonstrating its effectiveness in leveraging entity names from text.
  • Freebase benefits more: Textual augmentation yielded more significant improvements on Freebase than on Wordnet, likely due to the richer and more detailed Wikipedia descriptions available for Freebase entities.

Limitations and Future Work

  • Data quality dependency: The paper notes that the effectiveness of textual augmentation is highly dependent on the quality and relevance of the textual data. WordNet's definitions, while useful, were found to be less impactful than Wikipedia summaries for Freebase.
  • Unsupervised learning challenges: The authors acknowledge that unsupervised learning of word weights can sometimes lead to over-emphasis on irrelevant words early in training. They propose a phased training approach to mitigate this.
  • Future directions: The paper suggests exploring more sophisticated methods for combining textual and KG information, potentially by jointly learning word embeddings and KG embeddings from scratch or by investigating how to better leverage the semantic structure within text. Further research into how to automatically assess the quality of textual data for KG augmentation is also a promising avenue.

Summary updated

BibTeX
@inproceedings{veira2019unsupervised,
  author = {Neil Veira and Brian Keng and Kanchana Padmanabhan and Andreas G. Veneris},
  title = {{Unsupervised Embedding Enhancements of Knowledge Graphs using Textual Associations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/725},
}
Figure