HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs

Rui Zhang, PhD, Facmi, Famia, Fiahsi, Yongkang Xiao, Rui ZhangarXiv 2025 · DOI · arXiv

Abstract

Multimodal Knowledge Graph Completion (MMKGC) aims to infer missing facts in knowledge graphs by leveraging auxiliary information such as images and text. Traditional methods are primarily discriminative, scoring potential candidates within a closed-world setting. We introduce HERGC, a framework that adopts a "retrieve-and-generate" paradigm. HERGC utilizes a Heterogeneous Experts Representation Retriever (HERR) to identify candidate entities and a Generative LLM Predictor (GLP) to reason through these candidates using multimodal evidence. Our approach achieves state-of-the-art performance on MKG-W, MKG-Y, and DB15K benchmarks, demonstrating the power of grounding Large Language Models (LLMs) in multimodal graph data.

Introduction

Knowledge Graphs (KGs) represent factual information as triples, yet they are often incomplete. Multimodal KGs (MMKGs) incorporate images and text to provide richer context for link prediction. However, existing discriminative models are limited by their inability to leverage external world knowledge. HERGC addresses this by treating completion as a generative task, allowing LLMs to "reason" through missing links with high accuracy by evaluating a retrieved set of candidates.

Methodology

The HERGC framework consists of two primary stages: retrieval and generative prediction.

Heterogeneous Experts Representation Retriever (HERR)

The HERR module aligns diverse modalities—text, images, and graph structure—using a Mixture of Heterogeneous Experts (MoHE). This approach employs both simple linear and complex Block-Hypercomplex experts to capture intricate cross-modal interactions. To handle the varying relevance of modalities across different relations, we implement a Relation-aware Gated Multimodal Unit (RaGMU), which dynamically weights inputs based on the query relation.

The HERGC architecture, featuring the HERR retrieval stage and the GLP generative prediction stage.

Generative LLM Predictor (GLP)

The top-$k$ candidates from HERR are passed to an LLM. For open-source models like Llama-3, we employ Embedding Injection, where fused multimodal embeddings are directly integrated into the LLM's prompt space via a lightweight adapter. This grounding ensures the LLM's reasoning is informed by the specific visual and structural context of the graph.

Experimental Results

HERGC was evaluated on the MKG-W, MKG-Y, and DB15K datasets. It established new state-of-the-art records, notably improving Hits@1 by 7.44% on MKG-Y and 3.94% on MKG-W compared to previous best methods.

Ablation Study

Ablation experiments confirm the necessity of each component. Removing specific modalities or model components like the Relation-aware Gated Multimodal Unit (RaGMU) leads to a measurable decline in performance across all metrics, as detailed in the performance table below.

Ablation study results showing the impact of different modalities and model components on performance across three benchmarks.

Robustness and Visualization

The model exhibits high robustness, maintaining stable performance even when 30% of multimodal data is missing or noisy. Visualization via t-SNE reveals that HERGC's fused embeddings produce significantly cleaner clusters by entity type compared to unimodal representations.

Conclusion

HERGC demonstrates that a generative approach to KG completion, grounded in multimodal embeddings, significantly outperforms traditional discriminative methods. While the use of LLMs increases computational cost, the gains in accuracy and reasoning capability are substantial. Future work will focus on enhancing the extraction of relational clues from visual data and optimizing inference efficiency.

Summary updated

BibTeX
@misc{fiahsi2025hergc,
  author = {Rui Zhang, PhD, FACMI, FAMIA, FIAHSI and Yongkang Xiao and Rui Zhang},
  title = {{HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs}},
  year = {2025},
  eprint = {2506.00826},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.00826},
}
Figure