Abstract
Multi-modal Knowledge Graph Embeddings (KGE) leverage diverse data types—such as text, images, and graph structures—to create comprehensive representations of real-world entities. However, a critical challenge is spatial heterogeneity: different modalities exist in distinct mathematical spaces, and traditional fusion methods often destroy their inherent structural geometry. To address this, researchers introduce Optimal Transport Knowledge Graph Embeddings (OTKGE). By modeling multi-modal fusion as a distributional alignment problem using Optimal Transport, OTKGE achieves state-of-the-art performance in link prediction tasks while preserving the intrinsic geometry of the data.
The Challenge of Spatial Heterogeneity
Knowledge Graphs (KGs) are essential for representing real-world information, but they are frequently incomplete. KGE methods address this by representing entities and relations in low-dimensional spaces to predict missing links.
While early KGE methods relied solely on the symbolic structure of the graph, modern multi-modal approaches incorporate auxiliary information like textual descriptions and visual data. The primary hurdle in this approach is spatial heterogeneity. Textual, visual, and structural features each possess unique distributions and mathematical spaces. Traditional fusion techniques, such as vector concatenation or simple averaging, often disrupt the intrinsic geometry of these individual spaces. This disruption leads to noisy, inconsistent unified representations that fail to fully capture the richness of the multi-modal data.
Methodology: Optimal Transport for Fusion
To overcome spatial heterogeneity, the authors propose OTKGE. Instead of relying on basic vector operations, OTKGE models the fusion process as an Optimal Transport (OT) problem.
- Distributional Alignment: The features of each modality are treated as probability distributions. Using OT, the model calculates a "transport plan" that aligns linguistic and visual embeddings with the structural embedding space. This is achieved by minimizing the Wasserstein distance, which measures the minimum "work" required to transform one distribution into another.
- Wasserstein Barycenter: Once the distributions are aligned, the modalities are fused into a unified representation by computing the Wasserstein barycenter. This barycenter acts as a representative average distribution that successfully preserves the geometric properties of all input modalities.
- Relational Transformation: Finally, standard KGE transformations (such as TransE or RotatE) are applied to the unified entity representations to model relationships and train the network using a standard link prediction loss.
Key Insights and Theoretical Guarantees
The fundamental insight of OTKGE is treating multi-modal fusion as a geometric alignment problem. By minimizing the Wasserstein distance, the model ensures that entities sharing similarities in visual or textual spaces remain proximate in the unified space.
Furthermore, the authors provide theoretical guarantees that OT fusion ensures both consistency (the unified space respects the information of individual modalities) and comprehensiveness (the unified space outperforms any single modality). This approach effectively overcomes the "mean problem" seen in simple averaging, which tends to pull embeddings toward the center of the space and strip them of their discriminative power.
Experimental Results
The OTKGE model was evaluated on two established multi-modal benchmarks:
- WN9-IMG: A dataset derived from WordNet containing roughly 7,000 entities, 9 relations, images, and synset definitions.
- FB-IMG: A larger dataset derived from Freebase (FB15k) featuring 11,000 entities, 1,231 relations, textual descriptions, and visual data.
OTKGE achieved state-of-the-art results across both datasets. On the complex FB-IMG dataset, OTKGE reached a Mean Reciprocal Rank (MRR) of 0.843, significantly outperforming previous multi-modal leaders like MMKRL (0.827) and TBKGE (0.812).
Qualitatively, t-SNE visualizations demonstrate that OTKGE produces exceptionally clean semantic clusters. For instance, in the FB-IMG dataset, the model clearly separates distinct entity types (e.g., people, places, and events), avoiding the significant overlap and confusion typical of traditional structural-only or concatenation-based methods.
Limitations and Future Directions
While highly effective, the framework has a few limitations:
- Pre-trained Dependency: The quality of the final embeddings relies heavily on the initial features extracted by pre-trained models (e.g., VGG for images, GloVe for text). Poor initial features limit the effectiveness of the alignment.
- Computational Complexity: Solving Optimal Transport problems is computationally more expensive than simple vector addition, although the authors mitigate this by utilizing efficient barycenter strategies.
Future work may extend this framework to incorporate additional modalities, such as video or audio, and explore more complex base KGE models beyond TransE and RotatE.
Summary updated