Convolutional Neural Knowledge Graph Learning

Feipeng Zhao, Martin Renqiang Min, Chen Shen, Amit Chakraborty

TL;DR

This paper introduces a novel approach to knowledge graph learning by employing Convolutional Neural Networks (CNNs) to capture complex, non-linear relationships between entities and their associated relations. By treating triplets as matrices and applying CNNs, the model achieves state-of-the-art performance in link prediction, demonstrating the power of deep learning for understanding structured knowledge.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are powerful tools for representing real-world information, but they are often incomplete. A key challenge in KG research is to predict missing links or infer new relationships. Traditional methods, like those based on translation (e.g., TransE, TransH, TransR), model relationships as simple vector additions between entities. While effective, this "translation" assumption is a simplification and fails to capture the more intricate and nuanced connections that exist in real-world knowledge. This paper aims to address this limitation by developing a model that can learn more complex, interactive patterns between entities and relationships.

Methods

The core innovation of this paper is the application of Convolutional Neural Networks (CNNs) to knowledge graph learning. Instead of treating entities and relationships as simple vectors for translation, the authors represent them as low-dimensional sequential vectors. A knowledge graph triplet (head entity, relationship, tail entity) is then transformed into a 3xk matrix (where k is the embedding dimension). This matrix is fed into a CNN, which uses convolutional kernels to learn local features and complex interactions within the triplet.

Key steps in their CNN-based model include:

  • Combination: Merging the embedding vectors of the head, relationship, and tail into a single matrix.
  • Convolution Operation: Applying 3x3 kernels across the matrix to extract local patterns. Unlike image CNNs, these kernels primarily operate along the rows due to the matrix's structure.
  • Subsampling (Max-Pooling): Reducing the dimensionality of the feature maps to capture the most important information.
  • Fully Connection: Flattening the pooled features and passing them through a linear layer to create a high-level representation.
  • Logistic Regression: Using a sigmoid function on the final output to produce a confidence score for the triplet, ranging between 0 and 1.

The model is trained end-to-end using positive triplets from the KG and manually generated negative triplets (created by corrupting positive ones). A pairwise ranking loss function is employed to ensure that positive triplets receive higher scores than negative ones.

Datasets

The authors evaluated their model on two widely used public benchmark datasets for knowledge graph learning:

  • FB15k: Derived from the Google Knowledge Graph Freebase dataset, containing entities like people, places, and events, along with thousands of relationships.
  • WN18: Generated from WordNet, a large lexical database of English.

Key Insights and Ideas

  • Beyond Simple Translations: The paper convincingly argues that relationships in KGs are more complex than simple vector translations, and CNNs are well-suited to capture these intricate patterns.
  • Treating Triplets as Matrices: The novel idea of forming a matrix from triplet embeddings and applying CNNs is a significant departure from previous approaches and unlocks new representational power.
  • End-to-End Learning: Simultaneously learning entity/relationship embeddings and the CNN scoring function allows for a more cohesive and effective model.
  • CNNs for Structured Data: This work demonstrates the versatility of CNNs, extending their successful application from image and text processing to the domain of structured knowledge graphs.

Key Results

The proposed CNN model achieved state-of-the-art results on the FB15k and WN18 datasets for link prediction.

  • FB15k: The CNN model achieved a Hits@10 score of 94.5%, outperforming other methods. While its Mean Rank (68) was slightly worse than some, it was still competitive.
  • WN18: The model demonstrated excellent performance with a Mean Rank of 17, significantly outperforming other models which had Mean Ranks above 200. It also achieved a strong Hits@10 score of 96.2%.

In triplet classification, the CNN model also showed competitive results, outperforming other models on the Hits@1 metric for FB15k.

Limitations and Future Work

  • Negative Sampling: The paper acknowledges that manually corrupted negative triplets might sometimes represent actual valid relationships, potentially introducing noise into the training process. They mention that future work could explore more sophisticated sampling methods.
  • Binary Classification: The authors suggest that in the future, they could explore treating KG embedding learning as a binary classification problem, assigning a label of 1 to positive triplets and 0 to negative ones, and using different binary classification loss functions.
  • Relationship Paths: While their model captures complex interactions, the paper doesn't explicitly model "relation paths" as some other methods do, which could be a direction for future exploration.
  • Entity Descriptions: The paper notes that other works incorporate entity descriptions to improve performance, a feature not included in their current CNN model, suggesting it as a potential avenue for enhancement.

Summary updated

Abstract

Previous models for learning entity and relationship embeddings of knowledge graphs such as TransE, TransH, and TransR aim to explore new links based on learned representations. However, these models interpret relationships as simple translations on entity embeddings. In this paper, we try to learn more complex connections between entities and relationships. In particular, we use a Convolutional Neural Network (CNN) to learn entity and relationship representations in knowledge graphs. In our model, we treat entities and relationships as one-dimensional numerical sequences with the same length. After that, we combine each triplet of head, relationship, and tail together as a matrix with height 3. CNN is applied to the triplets to get confidence scores. Positive and manually corrupted negative triplets are used to train the embeddings and the CNN model simultaneously. Experimental results on public benchmark datasets show that the proposed model outperforms state-of-the-art models on exploring unseen relationships, which proves that CNN is effective to learn complex interactive patterns between entities and relationships.
BibTeX
@misc{zhao2017convolutional,
  author = {Feipeng Zhao and Martin Renqiang Min and Chen Shen and Amit Chakraborty},
  title = {{Convolutional Neural Knowledge Graph Learning}},
  year = {2017},
  eprint = {1710.08502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1710.08502},
}
Figure