IMF: Interactive Multimodal Fusion Model for Link Prediction

Xinhang Li, Xiangyu Zhao, Jiaxing Xu, Yong Zhang, Chunxiao Xing

TL;DR

This paper introduces the Interactive Multimodal Fusion (IMF) model, a novel two-stage framework for link prediction in multimodal knowledge graphs. IMF effectively integrates information from structural, visual, and textual modalities by first fusing them using a bilinear mechanism with contrastive learning, and then combining their individual predictions through a decision fusion stage, outperforming existing methods.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are powerful tools for representing information, but they are often incomplete, requiring link prediction to fill in missing relationships. While incorporating multimodal information (like images and text descriptions) has shown promise, existing methods often treat these modalities separately or project them into a single unified space. This approach can lead to a loss of modality-specific nuances and fails to fully capture the complementary nature of different data sources. The paper addresses this by proposing a method that better models the complex interactions between modalities, akin to how humans integrate diverse information.

Methods

The core of the IMF model is its two-stage fusion approach:

  1. Modality-Specific Encoders: Pre-trained encoders (GAT for structural, VGG16 for visual, and BERT for textual) are used to extract latent features for each modality. These encoders are kept fixed during training.
  2. Multimodal Fusion Module: This is the first fusion stage. Instead of a simple projection, IMF uses a bilinear pooling mechanism, inspired by Tucker decomposition, to model rich interactions between modalities. Crucially, it incorporates contrastive learning as a regularization technique. This contrastive loss encourages representations of the same entity across different modalities to be closer, while pushing representations of different entities further apart. This helps preserve modality-specific information while learning commonalities.
  3. Contextual Relational Model: For each modality (including the fused multimodal representation), a modality-specific scoring function is used to generate predictions. These models leverage relation information as context.
  4. Decision Fusion Module: This is the second fusion stage. Instead of relying solely on the fused multimodal representation, IMF combines the predictions from each individual modality and the fused multimodal representation. This weighted averaging allows the model to leverage the strengths of each modality and their complementarity for the final prediction.

Datasets

The paper evaluates IMF on four public multimodal KG datasets:

  • DB15K, FB15K, YAGO15K: These are sourced from MMKG and consist of structural triples, entity images, and entity descriptions extracted from Wikidata.
  • FB15K-237: A subset of FB15K, reusing its visual and textual features.

The datasets are split into 70% for training, 10% for validation, and 20% for testing.

Key Insights and Ideas

  • Beyond Unified Spaces: The paper argues that simply projecting multimodal features into a single space is suboptimal. IMF's two-stage approach, preserving modality-specific representations before and during fusion, is key to its success.
  • Bilinear Fusion for Richer Interactions: The use of bilinear pooling (inspired by Tucker decomposition) allows for more complex interactions between modality representations than simpler methods.
  • Contrastive Learning as a Regularizer: Applying contrastive learning between modalities is a novel way to enforce consistency and improve the quality of fused representations, especially when data is scarce.
  • Decision Fusion for Complementarity: Combining individual modality predictions with the fused representation in a final decision stage effectively leverages the complementary strengths of each data source.

Key Results

  • Significant Performance Gains: IMF consistently outperforms state-of-the-art monomodal and multimodal baselines across all tested datasets. For example, on DB15K, IMF achieves statistically significant improvements, with gains of up to 42% for MRR.
  • Effectiveness of Components: Ablation studies show that each component of IMF (multimodal fusion, decision fusion, and contrastive learning) contributes positively to the overall performance. Removing any one of these leads to a performance drop.
  • Generalization: The framework is shown to be generalizable, as replacing the underlying scoring function with standard models like TransE, ConvE, or TuckER still results in improved performance when integrated into the IMF framework.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Modality Integrity: IMF assumes the availability of all modalities. Future work could explore methods to handle missing modalities or design components to predict missing modalities.
  • Efficiency: The current bilinear fusion model can be computationally intensive. Developing more lightweight fusion models or optimizing existing ones would be beneficial.
  • Broader Applicability: Exploring the application of IMF to other multimodal tasks beyond link prediction is a natural extension.

Summary updated

Abstract

Link prediction aims to identify potential missing triples in knowledge graphs. To get better results, some recent studies have introduced multimodal information to link prediction. However, these methods utilize multimodal information separately and neglect the complicated interaction between different modalities. In this paper, we aim at better modeling the inter-modality information and thus introduce a novel Interactive Multimodal Fusion (IMF) model to integrate knowledge from different modalities. To this end, we propose a two-stage multimodal fusion framework to preserve modality-specific knowledge as well as take advantage of the complementarity between different modalities. Instead of directly projecting different modalities into a unified space, our multimodal fusion module limits the representations of different modalities independent while leverages bilinear pooling for fusion and incorporates contrastive learning as additional constraints. Furthermore, the decision fusion module delivers the learned weighted average over the predictions of all modalities to better incorporate the complementarity of different modalities. Our approach has been demonstrated to be effective through empirical evaluations on several real-world datasets. The implementation code is available online at https://github.com/HestiaSky/IMF-Pytorch.
BibTeX
@inproceedings{li2023imf,
  author = {Xinhang Li and Xiangyu Zhao and Jiaxing Xu and Yong Zhang and Chunxiao Xing},
  title = {{IMF: Interactive Multimodal Fusion Model for Link Prediction}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583554},
}
Figure