Large Model Empowered Streaming Speech Semantic Communications

Zhenzi Weng, Zhijin Qin, Geoffrey Ye LiIEEE Wireless Communications Letters 2025 · OpenReview

TL;DR

This paper introduces LSSC-ST, a novel system for real-time, multilingual speech translation using large AI models and semantic communication principles. By offloading heavy computation to edge servers and processing speech in a streaming fashion with dynamic segment adjustment, LSSC-ST significantly reduces latency while maintaining high translation quality across languages.

What Problem Does This Paper Address?

Traditional communication systems focus on transmitting data bit-by-bit, ignoring the actual meaning. Semantic communication aims to transmit information based on its meaning, leading to more efficient communication. However, applying this to speech translation faces several challenges:

  • Computational Burden: Extracting semantic features from speech, especially for complex tasks like translation, requires significant computational power, which is often unavailable on user devices.
  • Multilingual Support: Existing semantic communication systems are typically limited to a single language, hindering seamless global communication.
  • Latency: Processing an entire speech utterance before translation introduces significant delays, making real-time applications impractical.

This paper tackles these issues by proposing a system that leverages large AI models for semantic understanding, enables multilingual translation, and minimizes latency through streaming and edge computing.

Methods

The core of the LSSC-ST system is an edge-device collaborative architecture that splits the workload between local devices and powerful edge servers.

  1. Speech Compression (Local Device): A lightweight neural network (NN) compresses incoming speech segments into an intermediate representation. This is done concurrently with capturing the next speech segment, a key to reducing latency.
  2. Semantic Feature Extraction & Channel Encoding (Edge Server): The compressed representation is sent to an edge server. Here, a large, pre-trained speech model (Meta Seamless Communication) extracts rich semantic features. These features are then encoded into symbols for transmission over a wireless channel.
  3. Channel Decoding & Speech Prediction (Edge Server/Local Device): The received symbols are decoded back into semantic features on the edge server. These features are then used by a lightweight speech predictor on the local device to reconstruct the translated speech segment.

A crucial innovation is the dynamic speech segmentation algorithm. Instead of fixed-length segments, this algorithm adaptively adjusts the duration of each segment based on the semantic content and amplitude slope of the previous segment. This helps avoid processing silent or redundant parts and ensures smoother transitions between translated segments.

Datasets

The system is trained and evaluated using the FLEURS dataset. This dataset is notable for covering speech in 102 languages, making it suitable for multilingual tasks. The authors specifically focus on English, Chinese, and French for their experiments.

Key Insights and Ideas

  • Large Models as Semantic Hubs: Leveraging massive, pre-trained language models (like Meta Seamless Communication) is key to achieving robust, multilingual semantic understanding without requiring extensive retraining for each new language pair.
  • Edge-Device Synergy: Offloading computationally intensive semantic extraction to edge servers is essential for enabling complex AI tasks on resource-constrained local devices.
  • Streaming is King for Latency: Processing speech in small, continuous streams, rather than waiting for complete utterances, is fundamental to achieving low-latency communication.
  • Dynamic Segmentation for Fluency: Adapting segment duration based on speech content, rather than using fixed lengths, can improve translation quality and reduce perceived latency by avoiding unnecessary processing.

Key Results

  • High Translation Quality: The LSSC-ST system achieves BLASER 2.0 scores of over 3.0 in high SNR regimes, indicating good translation quality.
  • Reduced Latency: The dynamic speech segmentation approach reduces transmission latency by 14.3% to 19.3% compared to fixed segmentation, achieving an average latency of around 0.4 seconds across tested languages.
  • Outperforms Benchmarks: LSSC-ST demonstrates superior performance compared to a benchmark system that cascades semantic communication for speech-to-text translation with a text-to-speech pipeline.

Limitations and Future Work

  • Fixed vs. Dynamic Segmentation Quality: The paper notes that the LSSC-ST with fixed speech segmentation provides slightly superior translation quality (BLASER 2.0 scores) compared to the dynamic approach. This is attributed to the speech compressor and predictor being trained under fixed-duration segments. This suggests a potential trade-off between latency reduction and immediate translation quality that could be further explored.
  • Future Work: The authors suggest that the dynamic speech segmentation algorithm offers a promising solution for low-latency multilingual speech translation. Further research could focus on optimizing the dynamic segmentation parameters and exploring its impact on different types of speech and noise conditions.

Summary updated

BibTeX
@article{weng2025large,
  author = {Zhenzi Weng and Zhijin Qin and Geoffrey Ye Li},
  title = {{Large Model Empowered Streaming Speech Semantic Communications}},
  journal = {IEEE Wireless Communications Letters},
  year = {2025},
  url = {https://openreview.net/forum?id=7mS6Lf39rt},
}
Figure