Synchronous Multi-Modal Semantic Communication System With Packet-Level Coding

Yun Tian, Jingkai Ying, Zhijin Qin, Ye Jin, Xiaoming Tao

TL;DR

This paper introduces SyncSC, a novel system for synchronous multi-modal communication (specifically video and speech) that tackles the challenges of semantic and temporal alignment, and robust transmission over packet-loss networks. It achieves this by employing 3DMM coefficients for video semantics and text for speech, coupled with packet-level forward error correction (FEC) and a BERT-based text loss concealment module.

What Problem Does This Paper Address?

The increasing demand for rich, interactive applications like video conferencing and Extended Reality (XR) necessitates efficient and synchronized transmission of multimodal data (e.g., video and audio). While semantic communication has shown promise in reducing bandwidth by transmitting meaning rather than raw data, existing approaches struggle with:

  • Synchronization: Aligning semantic features from different modalities (like facial expressions and speech content) in both time and meaning is difficult, especially when encoders are designed independently.
  • Packet Loss: Network packet loss can lead to complete loss of semantic information, severely degrading the quality of reconstructed data. Traditional physical-layer semantic communication systems are not well-equipped to handle this.
  • Bandwidth Efficiency: Transmitting high-quality, synchronized multimodal data, particularly talking-face videos, consumes significant bandwidth.

This paper aims to bridge this gap by proposing a system that ensures synchronized multimodal communication with enhanced robustness against packet loss, all while minimizing bandwidth usage.

Methods

The proposed Synchronous Multi-modal Semantic Communication System (SyncSC) comprises a transmitter and a receiver with several key components:

  • Video Semantic Encoder: Instead of transmitting raw video frames or keypoints, SyncSC uses 3D Morphable Model (3DMM) coefficients to represent facial semantics. These coefficients capture expression, rotation, translation, and eye motion, offering a compact and semantically rich representation.
  • Speech Semantic Encoder: Speech is converted into text using an Automatic Speech Recognition (ASR) system. This text, along with timestamps, serves as the semantic representation for speech.
  • Packet-Level Forward Error Correction (FEC) - PacSC: To combat packet loss, a novel packet-level FEC mechanism called PacSC is introduced. It leverages semantic redundancy within video frames, treating encoded semantics as symbols and employing a Masked Autoencoder (MAE)-based approach for erasure coding. This is designed to be more effective than traditional FEC methods at high packet loss rates.
  • Text Packet Loss Concealment (TextPC): For text packets, a BERT-based module is proposed to predict lost words. This leverages the power of pre-trained language models to recover missing semantic information without significantly increasing bandwidth.
  • Image Generator: At the receiver, a generator reconstructs facial images from the decoded 3DMM coefficients, using a reference image as a base.
  • Visual-Guided Speech Synthesis: A key innovation is the visual-guided speech synthesis module. It uses synchronized facial semantics (3DMM coefficients) and text to guide the generation of lip-synchronized speech, ensuring temporal and semantic alignment between audio and video. This module incorporates lip encoders, content encoders, duration aligners, and audio generators.

Datasets

  • VoxCeleb dataset: Used for training and evaluating video semantic coding, packet coding, and the image generator. Videos are cropped to 256x256 and contain facial parts.
  • Chem dataset: Used for training and testing the visual-guided speech synthesis module. Videos are cropped from subtitle files to obtain sentence-level video clips.

Key Insights and Ideas

  • 3DMM Coefficients as Semantics: Using 3DMM coefficients for facial representation is a more efficient and semantically interpretable alternative to keypoints, especially for capturing subtle facial movements and expressions.
  • Packet-Level FEC for Semantics: Applying FEC at the semantic packet level, rather than at the bit or symbol level, is crucial for robust multimodal communication in lossy networks. The MAE-based approach for PacSC is a novel way to exploit semantic redundancy.
  • Visual Guidance for Speech Synthesis: Synchronizing speech synthesis with visual cues (facial expressions) is a powerful way to achieve natural-sounding and lip-synchronized audio, enhancing the overall user experience.
  • BERT for Text Recovery: Leveraging pre-trained language models like BERT for text packet loss concealment offers a significant advantage in recovering semantic meaning without substantial bandwidth overhead.

Key Results

  • Bandwidth Reduction: SyncSC achieves significant bandwidth savings compared to traditional codecs like H.264 and H.265 (e.g., ~84% and ~67% respectively) while maintaining comparable or better visual quality.
  • Robustness to Packet Loss: The proposed PacSC and TextPC modules demonstrate improved performance at high packet loss rates. For instance, TextPC provides a gain of 0.1 in BLEU and Semantic Similarity when packet loss exceeds 0.5, outperforming traditional RS codes.
  • Synchronization Performance: SyncSC achieves strong audio-visual synchronization, with LSE-D scores of 8.822 and LSE-C scores of 5.015, significantly outperforming baseline methods.
  • Speech Quality: The visual-guided speech synthesis module achieves good auditory quality, comparable to traditional methods, while consuming significantly less bandwidth.

Limitations and Future Work

  • Computational Complexity: The paper acknowledges that video generation and speech synthesis are computationally intensive, leading to end-to-end latency at the second level.
  • Future Work: The authors suggest optimizing latency through quantization and pruning of models. Further research could explore applying the lightweight video semantic packet-level coder to other semantic sources.

Summary updated

Abstract

Although the semantic communication with joint semantic-channel coding design has shown promising performance in transmitting data of different modalities over physical layer channels, the synchronization and packet-level forward error correction of multimodal semantics have not been well studied. Due to the independent design of semantic encoders, synchronizing multimodal features in both the semantic and time domains is a challenging problem. In this paper, we take the facial video and speech transmission as an example and propose a Synchronous Multimodal Semantic Communication System (SyncSC) with Packet-Level Coding. To achieve semantic and time synchronization, 3D Morphable Mode (3DMM) coefficients and text are transmitted as semantics, and we propose a semantic codec that achieves similar quality of reconstruction and synchronization with lower bandwidth, compared to traditional methods. To protect semantic packets under the erasure channel, we propose a packet-Level Forward Error Correction (FEC) method, called PacSC, that maintains a certain visual quality performance even at high packet loss rates. Particularly, for text packets, a text packet loss concealment module, called TextPC, based on Bidirectional Encoder Representations from Transformers (BERT) is proposed, which significantly improves the performance of traditional FEC methods. The simulation results show that our proposed SyncSC reduce transmission overhead and achieve high-quality synchronous transmission of video and speech over the packet loss network.
BibTeX
@article{tian2025synchronous,
  author = {Yun Tian and Jingkai Ying and Zhijin Qin and Ye Jin and Xiaoming Tao},
  title = {{Synchronous Multi-Modal Semantic Communication System With Packet-Level Coding}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2025},
  url = {https://openreview.net/forum?id=XWdGuFc1Ht},
}
Figure