TL;DR
This paper introduces SyncSC, a novel system for synchronous multi-modal communication (specifically video and speech) that tackles the challenges of semantic and temporal alignment, and robust transmission over packet-loss networks. It achieves this by employing 3DMM coefficients for video semantics and text for speech, coupled with packet-level forward error correction (FEC) and a BERT-based text loss concealment module.
What Problem Does This Paper Address?
The increasing demand for rich, interactive applications like video conferencing and Extended Reality (XR) necessitates efficient and synchronized transmission of multimodal data (e.g., video and audio). While semantic communication has shown promise in reducing bandwidth by transmitting meaning rather than raw data, existing approaches struggle with:
- Synchronization: Aligning semantic features from different modalities (like facial expressions and speech content) in both time and meaning is difficult, especially when encoders are designed independently.
- Packet Loss: Network packet loss can lead to complete loss of semantic information, severely degrading the quality of reconstructed data. Traditional physical-layer semantic communication systems are not well-equipped to handle this.
- Bandwidth Efficiency: Transmitting high-quality, synchronized multimodal data, particularly talking-face videos, consumes significant bandwidth.
This paper aims to bridge this gap by proposing a system that ensures synchronized multimodal communication with enhanced robustness against packet loss, all while minimizing bandwidth usage.
Methods
The proposed Synchronous Multi-modal Semantic Communication System (SyncSC) comprises a transmitter and a receiver with several key components:
- Video Semantic Encoder: Instead of transmitting raw video frames or keypoints, SyncSC uses 3D Morphable Model (3DMM) coefficients to represent facial semantics. These coefficients capture expression, rotation, translation, and eye motion, offering a compact and semantically rich representation.
- Speech Semantic Encoder: Speech is converted into text using an Automatic Speech Recognition (ASR) system. This text, along with timestamps, serves as the semantic representation for speech.
- Packet-Level Forward Error Correction (FEC) - PacSC: To combat packet loss, a novel packet-level FEC mechanism called PacSC is introduced. It leverages semantic redundancy within video frames, treating encoded semantics as symbols and employing a Masked Autoencoder (MAE)-based approach for erasure coding. This is designed to be more effective than traditional FEC methods at high packet loss rates.
- Text Packet Loss Concealment (TextPC): For text packets, a BERT-based module is proposed to predict lost words. This leverages the power of pre-trained language models to recover missing semantic information without significantly increasing bandwidth.
- Image Generator: At the receiver, a generator reconstructs facial images from the decoded 3DMM coefficients, using a reference image as a base.
- Visual-Guided Speech Synthesis: A key innovation is the visual-guided speech synthesis module. It uses synchronized facial semantics (3DMM coefficients) and text to guide the generation of lip-synchronized speech, ensuring temporal and semantic alignment between audio and video. This module incorporates lip encoders, content encoders, duration aligners, and audio generators.
Datasets
- VoxCeleb dataset: Used for training and evaluating video semantic coding, packet coding, and the image generator. Videos are cropped to 256x256 and contain facial parts.
- Chem dataset: Used for training and testing the visual-guided speech synthesis module. Videos are cropped from subtitle files to obtain sentence-level video clips.
Key Insights and Ideas
- 3DMM Coefficients as Semantics: Using 3DMM coefficients for facial representation is a more efficient and semantically interpretable alternative to keypoints, especially for capturing subtle facial movements and expressions.
- Packet-Level FEC for Semantics: Applying FEC at the semantic packet level, rather than at the bit or symbol level, is crucial for robust multimodal communication in lossy networks. The MAE-based approach for PacSC is a novel way to exploit semantic redundancy.
- Visual Guidance for Speech Synthesis: Synchronizing speech synthesis with visual cues (facial expressions) is a powerful way to achieve natural-sounding and lip-synchronized audio, enhancing the overall user experience.
- BERT for Text Recovery: Leveraging pre-trained language models like BERT for text packet loss concealment offers a significant advantage in recovering semantic meaning without substantial bandwidth overhead.
Key Results
- Bandwidth Reduction: SyncSC achieves significant bandwidth savings compared to traditional codecs like H.264 and H.265 (e.g., ~84% and ~67% respectively) while maintaining comparable or better visual quality.
- Robustness to Packet Loss: The proposed PacSC and TextPC modules demonstrate improved performance at high packet loss rates. For instance, TextPC provides a gain of 0.1 in BLEU and Semantic Similarity when packet loss exceeds 0.5, outperforming traditional RS codes.
- Synchronization Performance: SyncSC achieves strong audio-visual synchronization, with LSE-D scores of 8.822 and LSE-C scores of 5.015, significantly outperforming baseline methods.
- Speech Quality: The visual-guided speech synthesis module achieves good auditory quality, comparable to traditional methods, while consuming significantly less bandwidth.
Limitations and Future Work
- Computational Complexity: The paper acknowledges that video generation and speech synthesis are computationally intensive, leading to end-to-end latency at the second level.
- Future Work: The authors suggest optimizing latency through quantization and pruning of models. Further research could explore applying the lightweight video semantic packet-level coder to other semantic sources.
Summary updated