Large Speech Model Enabled Semantic Communication

Yun Tian, Zhijin Qin, Guocheng Lv, Ye Jin, Kaibin Huang, Zhu HanarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces LargeSC, a novel semantic communication system for speech that leverages large pre-trained language models (like Moshi) to achieve highly efficient and robust transmission. By converting speech into discrete tokens and employing an adaptive controller with unequal error protection, LargeSC significantly improves speech quality under packet loss while maintaining low bitrates, making it suitable for real-time applications.

What Problem Does This Paper Address?

Traditional speech semantic communication systems, often based on Joint Source-Channel Coding (JSCC), face limitations due to their task-specific designs and the need for extensive retraining for new applications. While large pre-trained models have shown remarkable success in various domains, their application to adaptive, low-bitrate speech transmission over lossy channels remains underexplored. The core challenge is to balance compression efficiency, speech quality, and latency, especially in the presence of packet loss, which can severely degrade communication. Existing methods struggle to retain rich prosodic information and often incur significant delays, hindering real-time deployment.

Methods

The proposed LargeSC system integrates several key components:

  • Mimi Speech Codec: This pre-trained model is adapted to convert speech into discrete tokens. This token-based representation is crucial for compatibility with large language models and enables efficient transmission. The codec uses Residual Vector Quantization (RVQ) to generate semantic and acoustic tokens.
  • Adaptive Controller with Unequal Error Protection (UEP): This module dynamically adjusts compression and error protection based on speech content importance and channel conditions (packet loss probability). It prioritizes critical semantic information and allocates redundancy selectively, ensuring better resilience to packet loss without sacrificing bandwidth unnecessarily. This is achieved through a learned importance map that guides token selection and repetition.
  • Finetuned Moshi Large Speech Model: A pre-trained Moshi model is fine-tuned using Low-Rank Adaptation (LoRA) for packet loss concealment. This autoregressive model leverages its generative capabilities to predict and reconstruct lost speech tokens, relying only on previously received data to maintain causality for real-time streaming.
  • Semantic Decoder: This component reconstructs the speech waveform from the predicted tokens.

Datasets

  • LibriSpeech: Used for training and fine-tuning the semantic encoder and the Moshi large speech model. The dataset was resampled to 24kHz.
  • Common Voice: Used as an unseen dataset for evaluating the generalization performance of LargeSC under different packet loss rates.

Key Insights and Ideas

  • Leveraging Large Models for Speech Semantics: The paper demonstrates that large pre-trained models, typically used for text and vision, can be effectively adapted for speech semantic communication, unlocking their rich semantic understanding for improved compression and reconstruction.
  • Adaptive Compression and UEP as a Synergistic Approach: Combining content-aware adaptive compression with unequal error protection is shown to be crucial for balancing bitrate and quality under varying channel conditions. The adaptive controller intelligently decides which tokens are essential and which can be discarded or repeated.
  • Causal Autoregressive Decoding for Real-Time: Using an autoregressive large model for packet loss concealment, which only relies on past information, is key to achieving low end-to-end latency suitable for real-time applications.
  • Token-Based Representation for LLM Compatibility: Converting speech into discrete tokens makes it directly compatible with the token-based input of large language models, simplifying integration and leveraging their strengths.

Key Results

  • High Compression Efficiency: LargeSC supports bandwidths ranging from 550 bps to 2.06 kbps, outperforming conventional baselines in speech quality at comparable or lower bitrates. For instance, at a VisQOL of 3.85, it saves significant bandwidth compared to AAC, Opus, and SoundStream.
  • Superior Speech Quality Under Packet Loss: The system demonstrates significantly better speech quality (measured by VisQOL and PLCMOS) compared to baselines like AAC, Opus, and SoundStream, especially under high packet loss rates (up to 30%). It maintains a more robust listening experience and better lexical intelligibility (lower WER).
  • Low End-to-End Latency: LargeSC achieves an end-to-end latency of approximately 460 ms, making it suitable for real-time communication scenarios.
  • Good Generalization: The model shows strong generalization capabilities to unseen speakers and recording conditions, as demonstrated on the Common Voice dataset.

Limitations and Future Work

  • Computational Requirements: The proposed method relies on a large generative model (around 7 billion parameters), which demands significant computational resources for inference. While LoRA helps, further optimization through quantization, pruning, and distillation is suggested for more accessible deployment.
  • End-to-End Latency: While improved over some methods, the latency is still influenced by the large model inference time and interleaving delays. Future work could focus on architectural optimizations and hybrid strategies to further reduce latency.
  • Adaptive Bitrate Fluctuations: The adaptive bitrate can fluctuate significantly (0.4-1.6 kbps), which might require careful management in certain network environments.
  • Future Work: The authors suggest exploring further latency reduction, model compression techniques, and expanding the system's applicability to other modalities or complex communication scenarios.

Summary updated

Abstract

Existing speech semantic communication systems mainly based on Joint Source-Channel Coding (JSCC) architectures have demonstrated impressive performance, but their effectiveness remains limited by model structures specifically designed for particular tasks and datasets. Recent advances indicate that generative large models pre-trained on massive datasets, can achieve outstanding performance arexhibit exceptional performance across diverse downstream tasks with minimal fine-tuning. To exploit the rich semantic knowledge embedded in large models and enable adaptive transmission over lossy channels, we propose a Large Speech Model enabled Semantic Communication (LargeSC) system. Simultaneously achieving adaptive compression and robust transmission over lossy channels remains challenging, requiring trade-offs among compression efficiency, speech quality, and latency. In this work, we employ the Mimi as a speech codec, converting speech into discrete tokens compatible with existing network architectures. We propose an adaptive controller module that enables adaptive transmission and in-band Unequal Error Protection (UEP), dynamically adjusting to both speech content and packet loss probability under bandwidth constraints. Additionally, we employ Low-Rank Adaptation (LoRA) to finetune the Moshi foundation model for generative recovery of lost speech tokens. Simulation results show that the proposed system supports bandwidths ranging from 550 bps to 2.06 kbps, outperforms conventional baselines in speech quality under high packet loss rates and achieves an end-to-end latency of approximately 460 ms, thereby demonstrating its potential for real-time deployment.
BibTeX
@misc{tian2025large,
  author = {Yun Tian and Zhijin Qin and Guocheng Lv and Ye Jin and Kaibin Huang and Zhu Han},
  title = {{Large Speech Model Enabled Semantic Communication}},
  year = {2025},
  eprint = {2512.04711},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.04711},
}
Figure