OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport

Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, Qingming HuangNeurIPS 2022 · OpenReview

Abstract

Multi-modal Knowledge Graph Embeddings (KGE) leverage diverse data types—such as text, images, and graph structures—to create comprehensive representations of real-world entities. However, a critical challenge is spatial heterogeneity: different modalities exist in distinct mathematical spaces, and traditional fusion methods often destroy their inherent structural geometry. To address this, researchers introduce Optimal Transport Knowledge Graph Embeddings (OTKGE). By modeling multi-modal fusion as a distributional alignment problem using Optimal Transport, OTKGE achieves state-of-the-art performance in link prediction tasks while preserving the intrinsic geometry of the data.

The Challenge of Spatial Heterogeneity

Knowledge Graphs (KGs) are essential for representing real-world information, but they are frequently incomplete. KGE methods address this by representing entities and relations in low-dimensional spaces to predict missing links.

While early KGE methods relied solely on the symbolic structure of the graph, modern multi-modal approaches incorporate auxiliary information like textual descriptions and visual data. The primary hurdle in this approach is spatial heterogeneity. Textual, visual, and structural features each possess unique distributions and mathematical spaces. Traditional fusion techniques, such as vector concatenation or simple averaging, often disrupt the intrinsic geometry of these individual spaces. This disruption leads to noisy, inconsistent unified representations that fail to fully capture the richness of the multi-modal data.

Methodology: Optimal Transport for Fusion

To overcome spatial heterogeneity, the authors propose OTKGE. Instead of relying on basic vector operations, OTKGE models the fusion process as an Optimal Transport (OT) problem.

  1. Distributional Alignment: The features of each modality are treated as probability distributions. Using OT, the model calculates a "transport plan" that aligns linguistic and visual embeddings with the structural embedding space. This is achieved by minimizing the Wasserstein distance, which measures the minimum "work" required to transform one distribution into another.
  2. Wasserstein Barycenter: Once the distributions are aligned, the modalities are fused into a unified representation by computing the Wasserstein barycenter. This barycenter acts as a representative average distribution that successfully preserves the geometric properties of all input modalities.
  3. Relational Transformation: Finally, standard KGE transformations (such as TransE or RotatE) are applied to the unified entity representations to model relationships and train the network using a standard link prediction loss.
The overall framework for OT fusion, showing how text, images, and structure are aligned into a unified representation.

Key Insights and Theoretical Guarantees

The fundamental insight of OTKGE is treating multi-modal fusion as a geometric alignment problem. By minimizing the Wasserstein distance, the model ensures that entities sharing similarities in visual or textual spaces remain proximate in the unified space.

Furthermore, the authors provide theoretical guarantees that OT fusion ensures both consistency (the unified space respects the information of individual modalities) and comprehensiveness (the unified space outperforms any single modality). This approach effectively overcomes the "mean problem" seen in simple averaging, which tends to pull embeddings toward the center of the space and strip them of their discriminative power.

Experimental Results

The OTKGE model was evaluated on two established multi-modal benchmarks:

  • WN9-IMG: A dataset derived from WordNet containing roughly 7,000 entities, 9 relations, images, and synset definitions.
  • FB-IMG: A larger dataset derived from Freebase (FB15k) featuring 11,000 entities, 1,231 relations, textual descriptions, and visual data.

OTKGE achieved state-of-the-art results across both datasets. On the complex FB-IMG dataset, OTKGE reached a Mean Reciprocal Rank (MRR) of 0.843, significantly outperforming previous multi-modal leaders like MMKRL (0.827) and TBKGE (0.812).

Qualitatively, t-SNE visualizations demonstrate that OTKGE produces exceptionally clean semantic clusters. For instance, in the FB-IMG dataset, the model clearly separates distinct entity types (e.g., people, places, and events), avoiding the significant overlap and confusion typical of traditional structural-only or concatenation-based methods.

t-SNE visualization of OTKGE embeddings demonstrating clear separation of distinct semantic clusters.

Limitations and Future Directions

While highly effective, the framework has a few limitations:

  • Pre-trained Dependency: The quality of the final embeddings relies heavily on the initial features extracted by pre-trained models (e.g., VGG for images, GloVe for text). Poor initial features limit the effectiveness of the alignment.
  • Computational Complexity: Solving Optimal Transport problems is computationally more expensive than simple vector addition, although the authors mitigate this by utilizing efficient barycenter strategies.

Future work may extend this framework to incorporate additional modalities, such as video or audio, and explore more complex base KGE models beyond TransE and RotatE.

Summary updated

Abstract

Multi-modal knowledge graph embeddings (KGE) have caught more and more attention in learning representations of entities and relations for link prediction tasks. Different from previous uni-modal KGE approaches, multi-modal KGE can leverage expressive knowledge from a wealth of modalities (image, text, etc.), leading to more comprehensive representations of real-world entities. However, the critical challenge along this course lies in that the multi-modal embedding spaces are usually heterogeneous. In this sense, direct fusion will destroy the inherent spatial structure of different modal embeddings. To overcome this challenge, we revisit multi-modal KGE from a distributional alignment perspective and propose optimal transport knowledge graph embeddings (OTKGE). Specifically, we model the multi-modal fusion procedure as a transport plan moving different modal embeddings to a unified space by minimizing the Wasserstein distance between multi-modal distributions. Theoretically, we show that by minimizing the Wasserstein distance between the individual modalities and the unified embedding space, the final results are guaranteed to maintain consistency and comprehensiveness. Moreover, experimental results on well-established multi-modal knowledge graph completion benchmarks show that our OTKGE achieves state-of-the-art performance.
BibTeX
@inproceedings{cao2022otkge,
  author = {Zongsheng Cao and Qianqian Xu and Zhiyong Yang and Yuan He and Xiaochun Cao and Qingming Huang},
  title = {{OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=gbXqMdxsZIP},
}
Figure