Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning

Jing Zhu, Yuhang Zhou, Shengyi Qian, Zhongmou He, Tong Zhao, Neil Shah, Danai Koutra

TL;DR

The authors introduce MM-GRAPH, the first comprehensive benchmark for multimodal graph learning that integrates high-resolution visual data alongside text and graph structure. Their extensive evaluation reveals that while adding visual information consistently yields a performance boost of over 6%, current specialized multimodal Graph Neural Networks (GNNs) often struggle to outperform well-tuned conventional baselines, highlighting a significant need for better cross-modal fusion strategies.

What Problem Does This Paper Address?

Graph machine learning has traditionally focused on two pillars: the structural connectivity of the graph and textual attributes (Text-Attributed Graphs or TAGs). However, in the real world—especially in e-commerce and social media—visual information is just as critical. For example, two products might have very different textual descriptions but look nearly identical, suggesting a strong relationship that a text-only model would miss.

Despite this, visual data is often neglected in graph benchmarks. Existing multimodal datasets frequently suffer from "link rot" (broken image URLs) or poor data quality. There is a clear gap in understanding how visual, textual, and structural information should be integrated to improve downstream tasks like link prediction and node classification.

Methods

The researchers developed a standardized evaluation framework to test various combinations of feature encoders and graph models.

  1. Feature Encoders: They compared "aligned" encoders (like CLIP and ImageBind), which map text and images into a shared embedding space, against "unaligned" combinations (like T5 for text and ViT or DINOv2 for images).
  2. Graph Models: They evaluated five representative GNNs, including conventional models (GCN, SAGE, BUDDY) and specialized multimodal GNNs (MMGCN, MGAT). For knowledge graphs, they tested multimodal Knowledge Graph Embeddings (KGEs) like MoSE and VISTA.
  3. Standardization: The benchmark provides a unified dataloader and evaluator to ensure fair comparisons across different architectures and tasks.

Datasets

MM-GRAPH introduces seven diverse datasets across three primary tasks:

  • Link Prediction (LP): Amazon-Sports, Amazon-Cloth, and Goodreads-LP. These involve predicting co-purchasing behavior or user preferences.
  • Node Classification (NC): Ele-fashion and Goodreads-NC, focused on predicting product or book categories.
  • Knowledge Graph Completion (KGC): MM-CoDEx-s and MM-CoDEx-m, which are enhanced versions of the CoDEx benchmark featuring curated visual entities from Wikipedia.
Figure 1. Visualization of our Multimodal Graph Benchmark (MM-GRAPH). All nodes of our benchmark have both visual and text features. (a) Amazon-Sports: The image and text come from the original image and title of the sports equipment. (b) Goodreads-LP: The images correspond to book covers. We do not show the text features of Goodreads-LP since the book description is very long. (c) Ele-fashion: The images and texts correspond to the original image and title of the fashion product, respectively.

Key Insights and Ideas

  • The "Alignment" Advantage: Feature alignment is crucial. Encoders like CLIP and ImageBind, which pre-align modalities in a shared space, consistently outperformed independent encoders. This suggests that bridging the semantic gap between vision and text before graph processing is more effective than trying to fuse them later.
  • The Late Fusion Trap: Surprisingly, specialized multimodal GNNs (MMGCN, MGAT) often performed worse than conventional GNNs like SAGE. The authors hypothesize that these specialized models rely too heavily on "late fusion"—processing each modality separately and only combining them at the final layer—which fails to capture complex cross-modal interactions during message passing.
  • Vision as a Differentiator: In tasks like the Goodreads book graph, visual features (book covers) were particularly powerful because they contain implicit information (like author style or OCR-readable text) that isn't always present in the metadata.
  • Multimodal > Unimodal: The most consistent finding was that multimodal models always beat unimodal ones. The synergy between text and vision provides a more holistic view of the node than either could alone.

Key Results

  • Performance Gains: Integrating multimodal features resulted in a performance improvement of more than 6% across various datasets compared to using text or images alone.
  • GNN Baselines: Conventional GNNs (specifically SAGE) using concatenated multimodal features emerged as the top performers on most benchmarks, often beating models specifically designed for multimodal data.
Comparison showing that conventional GNNs often outperform specialized multimodal GNNs on the MM-GRAPH datasets.
  • Impact of Modalities: As shown in the figure below, the combination of both modalities (purple) significantly outperforms single-modality baselines across different datasets.
Performance comparison of unimodal vs. multimodal features, highlighting the consistent gain from visual information.

Limitations and Future Work

The authors acknowledge that MM-GRAPH currently focuses on text and images. Future iterations could incorporate audio and video features, though open-source datasets for these in a graph context are scarce.

A major takeaway for the research community is the need for more sophisticated fusion strategies. Since simple concatenation with a standard GNN often beats complex multimodal architectures, there is significant room to develop new GNN layers that can perform "early" or "mid-stage" cross-modal alignment more effectively.

Summary updated

Abstract

Graph machine learning has made significant strides in recent years, yet the integration of visual information with graph structure and its potential for improving performance in downstream tasks remains an underexplored area. To address this critical gap, we introduce the Multimodal Graph Benchmark (MM-GRAPH), a pioneering benchmark that incorporates both visual and textual information into graph learning tasks. MM-GRAPH extends beyond existing text-attributed graph benchmarks, offering a more comprehensive evaluation framework for multimodal graph learning Our benchmark comprises seven diverse datasets of varying scales (ranging from thousands to millions of edges), designed to assess algorithms across different tasks in real-world scenarios. These datasets feature rich multimodal node attributes, including visual data, which enables a more holistic evaluation of various graph learning frameworks in complex, multimodal environments. To support advancements in this emerging field, we provide an extensive empirical study on various graph learning frameworks when presented with features from multiple modalities, particularly emphasizing the impact of visual information. This study offers valuable insights into the challenges and opportunities of integrating visual data into graph learning.
BibTeX
@inproceedings{zhu2025mosaic,
  author = {Jing Zhu and Yuhang Zhou and Shengyi Qian and Zhongmou He and Tong Zhao and Neil Shah and Danai Koutra},
  title = {{Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01326},
}
Figure