Multimodal semantic communication system based on graph neural networks

Xinran Ba, Xinguang Zhang, Shufeng Li, Jin Yuan, Jun HuIntelligence 2025 · DOI

TL;DR

This paper introduces BR-GG-DeepSC, a novel multimodal semantic communication system that leverages graph neural networks (GNNs) and knowledge graphs to fuse information from images and text. It significantly improves communication efficiency and robustness, especially in noisy environments, by reducing transmitted symbols and enhancing task performance in emotion analysis and visual question answering.

What Problem Does This Paper Address?

Modern communication systems are awash in multimodal data (text, images, audio, video), but current semantic communication approaches often struggle with single-modal limitations, leading to information loss and inadequate fusion. This is particularly problematic in bandwidth-constrained or noisy environments where accurately conveying meaning is crucial. Existing methods often fail to capture the intricate semantic relationships between different modalities, leading to suboptimal performance in tasks like emotion analysis or visual question answering. The paper aims to bridge this gap by developing a system that can effectively integrate and understand multimodal information, prioritizing meaning over raw data transmission.

Methods

The core of the proposed system, BR-GG-DeepSC, is its novel multimodal fusion approach based on graph neural networks. Key techniques include:

  • Dynamic Graph Construction: Instead of static graphs, this system builds graphs dynamically based on real-time cross-modal correlations and Shapley interaction values, enhanced by knowledge graph embeddings. This allows for adaptive representation learning.
  • GAT-GCN Synergy: A hybrid fusion layer combines Graph Attention Networks (GATs) for attention mechanisms and Graph Convolutional Networks (GCNs) for neighborhood aggregation. This approach captures both local and global dependencies, mitigating oversmoothing in deep GNNs while preserving structural semantics.
  • Multilayer Bidirectional Cross-Attention: This mechanism allows for deep, fine-grained semantic associations between image and text modalities by enabling interactions at multiple levels.
  • Shapley-value-based Dynamic Weight Allocation: This technique optimizes the contribution of different modalities to the overall semantic representation, enhancing robustness.
  • LSTM-based Semantic Correction Network: A Long Short-Term Memory (LSTM) network is employed to mitigate distortions caused by physical and semantic noise, improving the semantic quality of recovered signals.
  • Knowledge Graph Integration: External knowledge graphs are used to enhance semantic associations between modalities, aiding in understanding concepts and relationships.

Datasets

  • MVSA dataset: Used for multimodal emotion analysis. It comprises MVSA-single (5,129 text pairs) and MVSA-multiple (19,600 text pairs) datasets, containing image and text data with emotion tags.
  • VQAv2 dataset: Used for visual question answering tasks.

Key Insights and Ideas

  • Meaning over Bits: The paper strongly advocates for a shift from traditional bit-level accuracy to semantic understanding in communication, especially for multimodal data.
  • Graphs as a Natural Representation: Representing multimodal data as dynamic graphs allows for capturing complex inter-modal relationships that static methods miss.
  • Synergy of GNNs: Combining GAT and GCN effectively leverages both attention-based feature weighting and neighborhood aggregation for robust feature fusion.
  • Knowledge is Power: Integrating external knowledge graphs significantly boosts the system's ability to disambiguate semantics and understand nuanced relationships between modalities.
  • Robustness in Noise: The proposed system demonstrates remarkable resilience to noise, outperforming conventional methods significantly in low Signal-to-Noise Ratio (SNR) conditions.

Key Results

  • Emotion Analysis: The BR-GG-DeepSC model achieved superior accuracy in emotion analysis tasks across various channel conditions (AWGN, Rayleigh, Rician), especially outperforming conventional methods at low SNRs.
  • Visual Question Answering (VQA): Similar performance improvements were observed for VQA tasks.
  • Symbol Reduction: For image transmission, the proposed system reduced transmitted symbols by approximately 70% compared to conventional methods (12,544 vs. 41,718 symbols).
  • Overall Efficiency: The system transmits approximately 33% of the total symbols required by conventional methods, demonstrating significant efficiency gains.
  • Robustness: The system shows high robustness to semantic and physical noise, maintaining accuracy even in challenging channel conditions.

Limitations and Future Work

  • Text Transmission Symbol Increase: While image transmission sees a significant symbol reduction, text transmission shows an increase in transmitted symbols compared to conventional methods. However, this increase is accompanied by improved robustness.
  • Future Work: The paper suggests that the system's potential for practical deployment in resource-constrained environments is validated. Further research could explore extending this framework to other modalities (audio, video) and more complex real-world scenarios. The authors also highlight the importance of joint learning and knowledge mapping for enhancing semantic understanding.

Summary updated

Abstract

Current semantic communication systems primarily use single-modal data and face challenges such as intermodal information loss and insufficient fusion, limiting their ability to meet personalized demands in complex scenarios. To address these limitations, this study proposes a novel multimodal semantic communication system based on graph neural networks. The system integrates graph convolutional networks and graph attention networks to collaboratively process multimodal data and leverages knowledge graphs to enhance semantic associations between image and text modalities. A multilayer bidirectional cross-attention mechanism is introduced to mine fine-grained semantic relationships across modalities. Shapley-value-based dynamic weight allocation optimizes intermodal feature contributions. In addition, a long short-term memory-based semantic correction network is designed to mitigate distortion caused by physical and semantic noise. Experiments performed using multimodal tasks (emotion analysis and visual question answering) demonstrate the superior performance of the system. Under low signal-to-noise ratio conditions, the proposed BERT-ResNet and GCN–GAT enhanced deep semantic communication (BR-GG-DeepSC) model achieves higher accuracy than conventional methods, while reducing the total number of transmitted symbols to approximately 33% of that in conventional approaches. These results validate the robustness, efficiency, and potential of the proposed system for practical deployment in resource-constrained environments.
BibTeX
@inproceedings{ba2025multimodal,
  author = {Xinran Ba and Xinguang Zhang and Shufeng Li and Jin Yuan and Jun Hu},
  title = {{Multimodal semantic communication system based on graph neural networks}},
  booktitle = {Intelligence},
  year = {2025},
  doi = {10.20517/ir.2025.41},
}
Figure