Multi-Modal Self-Supervised Semantic Communication

Hang Zhao, Hongru Li, Dongfang Xu, Shenghui Song, Khaled B. LetaiefMeditcom 2025 · DOI · OpenReview

TL;DR

This paper introduces a novel two-stage training approach for multi-modal semantic communication systems. By leveraging multi-modal self-supervised learning for pre-training, it significantly reduces communication overhead during training while maintaining or improving performance on downstream tasks, paving the way for more efficient edge AI.

What Problem Does This Paper Address?

Semantic communication aims to transmit the meaning of data rather than the raw bits, promising significant reductions in communication overhead for applications like IoT and XR. However, current research often overlooks the training phase, which can be communication-intensive, especially in dynamic wireless environments where frequent retraining is needed. This is particularly challenging for multi-modal systems, where combining information from different sources (like images and depth data) can lead to redundant transmissions during training if not handled efficiently. The paper addresses this gap by proposing a method to minimize training communication costs in multi-modal semantic communication systems.

Methods

The proposed solution employs a two-stage training strategy:

  1. Multi-modal Self-Supervised Pre-training: This stage focuses on learning task-agnostic semantic features from multi-modal data without requiring labels. It uses a novel loss function that decouples representations to learn both shared (common across modalities) and unique (modality-specific) information. This is crucial because single-modal pre-training can lead to performance degradation in multi-modal scenarios by overemphasizing shared features.
  2. Joint Supervised Fine-tuning: After pre-training, both the encoders (at edge devices) and the decoder (at the central server) are fine-tuned using labeled data for the specific downstream task. This stage leverages the pre-trained features, accelerating convergence and further reducing communication needs compared to training from scratch.

The core innovation lies in the pre-training phase, which uses information-theoretic analysis to guide the design of a loss function that maximizes both shared and unique information, leading to more robust and efficient feature extraction.

Datasets

The paper evaluates its proposed method on the NYU Depth V2 (NYUD2) dataset. This dataset contains 1,449 RGB-Depth image pairs from 27 indoor scene categories, which are consolidated into 10 distinct classes for the experiments.

Key Insights and Ideas

  • Multi-modal pre-training is crucial for performance: Simply applying single-modal self-supervised learning to each modality independently leads to performance degradation in multi-modal downstream tasks.
  • Decoupling shared and unique information is key: A successful multi-modal pre-training strategy must learn both commonalities and distinct characteristics across different data modalities.
  • Self-supervised learning significantly reduces training overhead: By learning generalizable features in a label-free manner, the system can achieve faster convergence and require less communication during the fine-tuning phase.
  • Robustness to limited labels: The proposed method demonstrates strong performance even with only 50% of the label information, outperforming supervised baselines and showing stable training dynamics where other methods might struggle.

Key Results

  • The proposed method significantly reduces training-related communication overhead.
  • It achieves performance comparable to or exceeding existing supervised learning approaches.
  • Experimental results on the NYU Depth V2 dataset show faster test accuracy improvement compared to fully supervised methods.
  • With 50% of the label information, the proposed method substantially outperforms the supervised baseline.
  • The proposed method maintains stable training dynamics, even under challenging conditions where other methods like Barlow Twins experience issues.

Limitations and Future Work

  • Limitations: The paper focuses on AWGN channels and does not explicitly discuss performance in more complex or dynamic wireless environments beyond the general motivation. The specific trade-offs controlled by hyperparameters like Asha and Auni are mentioned but not deeply explored in terms of their impact across various scenarios.
  • Future Work: The authors suggest future research should focus on designing efficient coding schemes for dynamic network conditions and improving the system's scalability. They also note that their framework can reliably support existing advanced resource allocation frameworks.

Summary updated

Abstract

Semantic communication is emerging as a promising paradigm that focuses on the extraction and transmission of semantic meanings using deep learning techniques. While current research primarily addresses the reduction of semantic communication overhead, it often overlooks the training phase, which can incur significant communication costs in dynamic wireless environments. To address this challenge, we propose a multi-modal semantic communication system that leverages multi-modal self-supervised learning to enhance task-agnostic feature extraction. The proposed approach employs self-supervised learning during the pre-training phase to extract task-agnostic semantic features, followed by supervised fine-tuning for downstream tasks. This dual-phase strategy effectively captures both modality-invariant and modality-specific features while minimizing training-related communication overhead. Experimental results on the NYU Depth V2 dataset demonstrate that the proposed method significantly reduces training-related communication overhead while maintaining or exceeding the performance of existing supervised learning approaches. The findings underscore the advantages of multi-modal self-supervised learning in semantic communication, paving the way for more efficient and scalable edge inference systems.
BibTeX
@inproceedings{zhao2025multi,
  author = {Hang Zhao and Hongru Li and Dongfang Xu and Shenghui Song and Khaled B. Letaief},
  title = {{Multi-Modal Self-Supervised Semantic Communication}},
  booktitle = {Meditcom},
  year = {2025},
  doi = {10.1109/meditcom64437.2025.11104414},
}
Figure