Zero-Shot Relational Learning for Multimodal Knowledge Graphs

Rui Cai, Shichao Pei, Xiangliang Zhang

Abstract

This paper introduces Multimodal Relation Extrapolation (MRE), the first framework designed for zero-shot relational learning in Multimodal Knowledge Graphs (MMKGs). By fusing entity images, text descriptions, and graph structures through a generative adversarial approach, MRE can accurately predict triples for entirely new relations that have no prior training examples. Experimental results across three benchmarks demonstrate that MRE significantly outperforms existing state-of-the-art methods, particularly in its ability to generalize to unseen relations.

1. Introduction

Knowledge Graphs (KGs) are essential for organizing information, but they are notoriously incomplete. While traditional Knowledge Graph Completion (KGC) focuses on predicting missing links for known relations, real-world graphs are dynamic—new relations emerge constantly. The challenge is "zero-shot" relational learning: predicting a triple like (The Aviator, Costume designer, Sandy Powell) when the relation Costume designer has never appeared in training data. Existing methods either require few-shot samples or rely solely on text, ignoring the rich visual and structural information present in modern Multimodal KGs.

2. Methodology

The MRE framework consists of three primary modules designed to leverage the multimodal nature of modern KGs for zero-shot extrapolation.

The MRE training pipeline, featuring the Multimodal Learner, Structure Consolidator, and Relation Embedding Generator.

2.1 Multimodal Learner

Instead of simple feature concatenation, MRE uses a joint Transformer encoder with a mask-and-reconstruct strategy. By randomly hiding parts of images or words in text and forcing the model to predict them, MRE learns fine-grained semantic alignments between different modalities, such as linking the word "film" to a movie poster image.

2.2 Structure Consolidator

This module employs a Graph Neural Network (GNN) to capture the topological context of entities. It aggregates information from an entity's neighbors to refine its representation, ensuring that the entity's position in the graph is integrated with its visual and textual features.

2.3 Relation Embedding Generator

To handle zero-shot scenarios, MRE utilizes a Generative Adversarial Network (GAN). A generator takes the text description of a new relation and produces a synthetic relation embedding, while a discriminator distinguishes these from real embeddings of seen relations. This allows the model to map abstract text descriptions into the high-dimensional embedding space used for link prediction.

3. Experiments and Results

The researchers evaluated MRE on zero-shot versions of three established multimodal benchmarks: FB15K-237-ZS, DB15K-ZS, and WN18-IMG-ZS. In these datasets, relations are split into "seen" (for training) and "unseen" (for testing zero-shot performance).

Comparison of MRE and IMF+ZSGAN across different ratios of unseen relations, showing MRE's superior zero-shot capability.

MRE significantly outperformed state-of-the-art baselines across all datasets. On FB15K-237-ZS, MRE achieved a 28% improvement in Hits@1 compared to the strongest baseline. The model also demonstrated remarkable robustness; even when the proportion of unseen relations was increased to 60%, MRE maintained superior performance over competing multimodal models like IMF+ZSGAN.

4. Key Insights

  • Multimodality as a Bridge: Visual and textual information provides a semantic "anchor" that remains consistent even when the graph structure for a new relation is non-existent.
  • Fine-Grained Alignment: The mask-and-reconstruct approach forces the model to understand the latent correlation between different modalities, such as a movie's visual aesthetic and its textual genre.
  • GANs for Extrapolation: Using a GAN to "hallucinate" relation embeddings from text descriptions effectively bridges the gap between natural language and the geometric space of the knowledge graph.

5. Conclusion

MRE represents a significant step forward in making Multimodal Knowledge Graphs more adaptable to dynamic, real-world environments. While currently limited by using a single image per entity, future work aims to incorporate multi-image support to further enhance the model's representative power.

Summary updated

BibTeX
@inproceedings{cai2024zero,
  author = {Rui Cai and Shichao Pei and Xiangliang Zhang},
  title = {{Zero-Shot Relational Learning for Multimodal Knowledge Graphs}},
  booktitle = {IEEE Big Data},
  year = {2024},
  doi = {10.1109/bigdata62323.2024.10825189},
}

Loading related content...

Figure