Scalable multi-modal representation learning networks

Zihan Fang, Ying Zou, Shiyang Lan, Shide Du, Yanchao Tan, Shiping Wang

Abstract

Multi-modal representation learning (MMRL) is a critical task in modern data science, aimed at fusing information from diverse sources like images and text. Traditional graph-based approaches, while effective at modeling relationships, often overlook high-order dependencies and struggle with the "out-of-sample" problem, where new data requires costly graph reconstruction. This paper introduces Scalable Multi-modal Representation Learning Networks (SMMRL), a framework that leverages hypergraph theory and a proximal operator-inspired neural network architecture. SMMRL preserves complex multi-way relationships and enables efficient inductive learning by processing new samples through learned projection matrices. Evaluations on six benchmark datasets show that SMMRL consistently outperforms state-of-the-art methods in both accuracy and computational efficiency.

1. Introduction

In the field of multi-modal representation learning, the goal is to map data from different modalities into a shared latent space where semantic relationships are preserved. Graph-based methods have become a standard tool for this task due to their ability to model data manifolds. However, these methods typically rely on standard graphs that only capture pairwise connections, ignoring the richer, high-order relationships inherent in complex datasets. Furthermore, many graph-based models are transductive, meaning they cannot easily generalize to unseen data without retraining on the entire augmented dataset.

2. Methodology

The SMMRL framework addresses these challenges by integrating hypergraph embedding with a novel neural network design that unrolls mathematical optimization steps into trainable layers.

The SMMRL framework architecture, illustrating the flow from multi-modal data to a consensus representation via projection matrices and row-sparsity constraints.

2.1 Hypergraph Embedding and Feature Selection

To capture high-order relationships, SMMRL constructs probabilistic hypergraphs for each modality. Unlike standard graphs, a hyperedge in a hypergraph can connect multiple nodes simultaneously, preserving the multi-way geometric structure of the data. To mitigate the impact of noise in high-dimensional features, the model incorporates a row-sparsity constraint using an $L_{2,1}$-norm penalty on the projection matrices. This mechanism effectively performs automatic feature selection by zeroing out irrelevant dimensions across the dataset.

2.2 Optimization-Inspired Neural Network

A distinguishing feature of SMMRL is its architecture, which is inspired by proximal operators used in traditional mathematical optimization. By unrolling the iterative steps of an optimization algorithm into the layers of a feedforward neural network, the model combines the theoretical guarantees of rigorous mathematics with the training efficiency and backpropagation capabilities of deep learning.

2.3 Inductive Learning and Scalability

SMMRL is designed as an inductive framework. Instead of learning static node embeddings, it learns a set of modality-specific projection matrices. This approach allows the model to process new, out-of-sample data points through simple matrix multiplication, avoiding the need for expensive graph reconstruction and retraining. This makes SMMRL highly suitable for dynamic, large-scale applications.

3. Experimental Evaluation

The performance of SMMRL was validated on six real-world multi-modal datasets: BDGP, ESP-Game, Flickr, HW, NUS-WIDE, and Reuters. These datasets cover various modalities, including visual features, biological annotations, and multilingual documents.

3.1 Key Results

SMMRL demonstrated superior performance across all benchmarks. For instance, on the BDGP dataset, it achieved an accuracy of 98.39%, outperforming the best inductive baseline by nearly 3%. The model also showed remarkable robustness, achieving near-peak performance even when the training set was reduced to 30% of the total data. In terms of efficiency, SMMRL's pre-calculation of hypergraph Laplacians and its projection-based inference led to significantly faster training and testing times compared to dynamic graph-based competitors.

4. Discussion and Insights

The success of SMMRL can be attributed to several key design choices:

  • High-Order Modeling: Ablation studies confirmed that hypergraphs provide a 2-3% accuracy boost over standard pairwise graphs by capturing essential semantic context.
  • Sparsity Constraints: The row-sparsity penalty was vital for handling high-dimensional datasets like Reuters, where other models often failed due to memory exhaustion.
  • Dynamic Modality Fusion: Rather than simple concatenation, SMMRL employs a trainable fusion layer that dynamically weights each modality, ensuring that noisy sources do not degrade the overall representation quality.

5. Conclusion

SMMRL represents a significant advancement in scalable multi-modal learning. By combining the strengths of hypergraph theory and optimization-inspired neural networks, it provides a robust, efficient, and highly accurate framework for mapping diverse data types into a shared space. Future research could extend this work to handle incomplete data modalities or integrate self-supervised learning to further enhance representation quality.

Summary updated

Abstract

Multi-modal representation learning is recognized for its comprehensive interpretation across diverse modalities. Although existing approaches have yielded favorable results, they face challenges in high-order information preservation and out-of-sample data generalization. To tackle these issues, we propose a scalable multi-modal representation learning networks framework, which aims to learn optimal modality-specific projection matrices to project multi-modal features to a shared representation space. Specifically, weight guided modality-wise and row-sparsity driven feature-wise measures are considered to achieve adaptively hierarchical feature selection from the original data. Then, within the unified latent representation space, we employ hypergraph embedding to preserve the intricate high-order local geometric structures within the modality-specific high-dimensional spaces. Finally, we propose a proximal operator-inspired network architecture to resolve the optimization objectives, streamlining the process of feature auto-weighted selection and representation learning. The experimental results highlight the effectiveness and superiority of the proposed method, while online testing on out-of-sample data further demonstrates robust generalization. The code of the proposed method is publicly available at: https://github.com/ZihanFang11/SMMRL.
BibTeX
@article{fang2025scalable,
  author = {Zihan Fang and Ying Zou and Shiyang Lan and Shide Du and Yanchao Tan and Shiping Wang},
  title = {{Scalable multi-modal representation learning networks}},
  journal = {Artif. Intell. Rev.},
  year = {2025},
  doi = {10.1007/s10462-025-11224-8},
}
Figure