Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks

Jiafan Li, Jiaqi Zhu, Liang Chang, Yilin Li, Miaomiao Li, Yang Wang, Yi Yang, Hongan WangIJCAI 2025 · DOI

TL;DR

This paper introduces HGNN-IMA, a novel method for node classification in multi-modal heterogeneous networks (MMHNs). It tackles the challenge of effectively fusing information from different modalities (like text and images) by learning their mutual influence during graph propagation, leading to improved classification accuracy.

What Problem Does This Paper Address?

Many real-world online platforms, such as movie or product review sites, can be modeled as multi-modal heterogeneous networks (MMHNs). These networks consist of various types of nodes (e.g., movies, actors, directors) connected by different types of relationships. Crucially, nodes in these networks often possess information from multiple modalities, such as text descriptions and images. Accurately classifying these nodes is vital for understanding the entities they represent.

Existing approaches to multi-modal fusion in graph neural networks (GNNs) often fall into two categories: early fusion (combining modalities at the beginning, risking loss of individual characteristics) or late fusion (fusing modalities at the end, potentially missing cross-modal guidance during propagation). This paper addresses the gap in effectively capturing the mutual influence between modalities throughout the entire information propagation process within MMHNs, especially when dealing with missing or misaligned modalities.

Methods

The proposed model, HGNN-IMA, is built upon the heterogeneous graph transformer (HGT) framework and consists of three main modules:

  1. Pre-processing Module: Encodes features from each modality (e.g., text, vision) using pre-trained encoders.
  2. Heterogeneous Network Propagation Module: This is the core innovation. It employs a "Cross-modal Influence Unit" that integrates a nested inter-modal attention mechanism into the standard inter-node attention of HGT.
  3. Nested Inter-modal Attention: Instead of just considering a node's own features or its neighbors' features within a single modality, this mechanism computes attention scores by considering the similarity across all modalities. This allows the model to adaptively learn how much influence one modality should have on another during the propagation process.
  4. Modality Alignment: An additional modulation term is introduced to encourage nodes that are similar across multiple modalities to have a stronger influence on each other, mitigating the impact of misaligned modalities.
  5. Attention Loss for Missing Modalities: A specific loss function is designed to penalize high attention scores for missing modalities, ensuring they don't unduly influence the learning process.
  6. Feature Fusion Module: Fuses the enriched representations from different modalities using a modality-level attention mechanism to produce the final node embeddings.

The training objective combines a standard cross-entropy loss with the attention loss for missing modalities.

Datasets

The paper evaluates HGNN-IMA on five benchmark datasets:

  • DOUBAN: Movie network data.
  • IMDB: Movie network data.
  • AMAZON: Product review network data.
  • AMAZON-1 & AMAZON-2: Larger, self-constructed datasets from the Electronics category of Amazon, with AMAZON-2 including price as a third modality.

Key Insights and Ideas

  • Mutual Influence is Key: The core insight is that modalities don't just exist in parallel; they actively influence each other during information propagation in MMHNs. Capturing this dynamic interaction is crucial for accurate node classification.
  • Nested Attention for Adaptive Fusion: The novel nested inter-modal attention mechanism is a significant contribution. It allows the model to learn how modalities should influence each other, rather than relying on fixed early or late fusion strategies.
  • Handling Imperfect Data: The explicit inclusion of mechanisms to address missing and misaligned modalities (attention loss and modulation) is vital for real-world applicability, where data is rarely perfect.
  • Beyond Simple Aggregation: The paper demonstrates that simply aggregating features from different modalities isn't enough. The way these features are aggregated, considering their inter-modal relationships, is what drives performance gains.

Key Results

HGNN-IMA consistently outperforms several state-of-the-art baselines across all five datasets on both Micro-F1 and Macro-F1 metrics.

  • The model achieves gains of 1.2% to 1.6% in Macro-F1 on the first three datasets and around 2.5% on the larger datasets.
  • Ablation studies confirm the effectiveness of each proposed component: the Cross-modal Influence Unit, modality alignment, and the attention loss for missing modalities all contribute to the overall performance.
  • The model demonstrates good stability, with standard deviation tests showing significant differences compared to strong baselines like HGT.
  • Hyperparameter analysis suggests an optimal number of propagation layers (K=3) for balancing information exchange and avoiding over-smoothing.

Limitations and Future Work

  • Computational Complexity: The paper notes that the total computational complexity is O(|V|² · |M|²), which might be a concern for very large networks with many modalities.
  • Future Work: The authors suggest extending the method to other tasks that require node representation learning with network structures. They also imply that further exploration into the nuances of defining cross-modal influence and its complexity could be beneficial.

Summary updated

BibTeX
@inproceedings{li2025representation,
  author = {Jiafan Li and Jiaqi Zhu and Liang Chang and Yilin Li and Miaomiao Li and Yang Wang and Yi Yang and Hongan Wang},
  title = {{Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/338},
}
Figure