Generative Video Semantic Communication via Multimodal Semantic Fusion with Large Model

Hang Yin, Li Qiao, Yu Ma, Shuo Sun, Kan Li, Zhen Gao, Dusit Niyato

Abstract

Despite significant advancements in traditional syntactic communications based on Shannon's theory, these methods struggle to meet the requirements of 6G immersive communications, especially under challenging transmission conditions. With the development of generative artificial intelligence (GenAI), progress has been made in reconstructing videos using high-level semantic information. In this paper, we propose a scalable generative video semantic communication framework that extracts and transmits semantic information to achieve high-quality video reconstruction. Specifically, at the transmitter, description and other condition signals (e.g., first frame, sketches, etc.) are extracted from the source video, functioning as text and structural semantics, respectively. At the receiver, the diffusion-based GenAI large models are utilized to fuse the semantics of the multiple modalities for reconstructing the video. Simulation results demonstrate that, at an ultra-low channel bandwidth ratio (CBR), our scheme effectively captures semantic information to reconstruct videos aligned with human perception under different signal-to-noise ratios. Notably, the proposed "First Frame+Desc." scheme consistently achieves CLIP score exceeding 0.92 at CBR = 0.0057 for SNR > 0 dB. This demonstrates its robust performance even under low SNR conditions.

Introduction

Traditional video compression standards, such as H.264 and H.265, are rooted in Shannon’s information theory, focusing on the accurate reconstruction of pixels. While effective in high-bandwidth scenarios, these methods suffer from a "cliff effect": when the signal-to-noise ratio (SNR) drops below a critical threshold, the video quality collapses into unwatchable artifacts. As 6G applications move toward immersive VR/AR, the demand for massive bandwidth and resilience to noise grows.

This paper introduces a Generative Video Semantic Communication (GVSC) framework designed to reconstruct high-quality video from ultra-sparse data. By shifting the focus from pixel-level accuracy to semantic meaning, the framework leverages large-scale generative AI (GenAI) to "re-imagine" video content at the receiver based on multimodal semantic hints transmitted from the source.

System Architecture

The proposed GVSC framework consisting of a transmitter for multimodal semantic extraction and a receiver utilizing GenAI for video reconstruction.

The GVSC framework splits the communication task between a transmitter and a receiver using a multimodal approach:

  1. Multimodal Semantic Extraction (Transmitter): The system extracts two types of semantics. Textual Semantics are generated using a large vision-language model (Video-LLaVA) to describe actions and environments. Visual Semantics provide structural anchors, such as a sequence of sketches (extracted via PiDiNet) or the first RGB frame.
  2. Adaptive Transmission: Textual data is sent using robust traditional coding (Turbo codes + QAM) to prevent scene hallucination. Visual data is transmitted via Deep Joint Source-Channel Coding (DJSCC) with an attention mechanism to prioritize critical structural components.
  3. Generative Reconstruction (Receiver): The receiver utilizes pre-trained diffusion models, such as Open-Sora or Stable Diffusion, to fuse the received multimodal semantics. The model performs guided denoising to generate a fluid video that maintains high semantic alignment with the original source.

To ensure visual quality, the authors employ a weighted perceptual loss function combining Mean Squared Error (MSE) and Learned Perceptual Image Patch Similarity (LPIPS), optimizing for human perception rather than just mathematical error.

Experimental Results

The framework was evaluated using the WebVid dataset at ultra-low Channel Bandwidth Ratios (CBR) as low as 0.0031. A primary advantage of GVSC is its graceful degradation under noise compared to traditional codecs.

Qualitative comparison of video reconstruction across different methods at SNR = 10 dB and SNR = 0 dB.

As illustrated in Figure 2, at an SNR of 0 dB—where noise is as strong as the signal—traditional H.264 and H.265 codecs fail completely. In contrast, the proposed GVSC variants, particularly the "First Frame + Desc." scheme, produce clear and visually pleasing videos. Quantitatively, this scheme achieves a CLIP score exceeding 0.92 for SNR > 0 dB at a CBR of 0.0057, indicating exceptional alignment between the reconstructed video and the original semantic intent.

Conclusion

GVSC demonstrates that transmitting the "concept" and "structure" of a video is significantly more efficient than transmitting raw pixels in resource-constrained environments. While the framework offers high resilience to noise and avoids the traditional cliff effect, challenges remain regarding the computational complexity of diffusion models and maintaining fine-grained temporal consistency over long video durations. Future research will focus on optimizing these generative models for real-time, low-latency 6G applications.

Summary updated

Abstract

Despite significant advancements in traditional syntactic communications based on Shannon's theory, these methods struggle to meet the requirements of 6G immersive communications, especially under challenging transmission conditions. With the development of generative artificial intelligence (GenAI), progress has been made in reconstructing videos using high-level semantic information. In this paper, we propose a scalable generative video semantic communication framework that extracts and transmits semantic information to achieve high-quality video reconstruction. Specifically, at the transmitter, description and other condition signals (e.g., first frame, sketches, etc.) are extracted from the source video, functioning as text and structural semantics, respectively. At the receiver, the diffusion-based GenAI large models are utilized to fuse the semantics of the multiple modalities for reconstructing the video. Simulation results demonstrate that, at an ultra-low channel bandwidth ratio (CBR), our scheme effectively captures semantic information to reconstruct videos aligned with human perception under different signal-to-noise ratios. Notably, the proposed ``First Frame+Desc." scheme consistently achieves CLIP score exceeding 0.92 at CBR = 0.0057 for SNR > 0 dB. This demonstrates its robust performance even under low SNR conditions.
BibTeX
@article{yin2025generative,
  author = {Hang Yin and Li Qiao and Yu Ma and Shuo Sun and Kan Li and Zhen Gao and Dusit Niyato},
  title = {{Generative Video Semantic Communication via Multimodal Semantic Fusion with Large Model}},
  journal = {IEEE Transactions on Vehicular Technology},
  year = {2025},
  doi = {10.1109/tvt.2025.3595688},
}
Figure