Robust Semantic Communications for Speech Transmission

Zhenzi Weng, Zhijin Qin, Geoffrey Ye LiICASSP 2025 · OpenReview

TL;DR

This paper introduces Ross-S2T, a novel semantic communication system for speech-to-text translation (S2TT) that focuses on transmitting the core meaning of speech rather than raw audio bits. It tackles the challenge of corrupted speech by employing a generative adversarial network (GAN) to recover lost semantic information and a "probe" mechanism to pinpoint and correct semantic errors, achieving superior S2TT performance and robustness.

What Problem Does This Paper Address?

Traditional communication systems transmit data at the bit level, which is inefficient and prone to errors, especially with noisy or corrupted signals. Semantic communication aims to overcome this by transmitting the meaning of the information, leading to more efficient and robust communication. While semantic communication has shown promise for text and image transmission, its application to speech, particularly for tasks like speech-to-text translation (S2TT) with corrupted input, remains a significant challenge. Existing methods often extract shallow semantic features or require intermediate text generation, hindering end-to-end optimization. Furthermore, the impact of semantic impairments (like background noise or interference) on speech communication systems hasn't been thoroughly addressed. This paper tackles these issues by proposing a system that can perform S2TT robustly even when the input speech is corrupted.

Methods

The proposed system, Ross-S2T, operates in two main stages:

  1. DeepSC-S2T (Clear Speech Training): This initial stage focuses on learning to extract deep semantic features from clear speech and translate them into the target language. It uses a novel "semantic extractor" and "semantic converter" to transform source speech into deep semantic features. These features are then encoded, transmitted, and decoded to produce the target text. An end-to-end loss function, Label Smoothing Regularization-aided Cross-Entropy (LSR-CE), is used to train this stage, promoting better prediction accuracy.
  2. Ross-S2T (Corrupted Speech Training & Testing): This stage introduces a "dual-compensator" mechanism to handle corrupted speech.
  3. GAN-enabled Deep Semantic Compensator: A Generative Adversarial Network (GAN) is employed at the transmitter. The generator part of the GAN is trained to take corrupted speech and produce compensated deep semantic features that closely resemble the features extracted from clear speech. The discriminator learns to distinguish between real (clear speech derived) and fake (corrupted speech derived) semantic features.
  4. Semantic Impairment Probe-aided Compensator: To further refine the compensated features, a "semantic impairment probe" is generated. This probe, essentially an index vector, identifies the locations of corrupted semantic features. This information is then used by a probe-aided compensator at the receiver to precisely calibrate the semantic features, reducing ambiguity and improving the understandability of the final target text.

Datasets

  • CoVOST 2: Used as the clear speech dataset for training.
  • Corrupted Speech Dataset: Created by adding background noise and external speech interference to the clear speech from CoVOST 2.

Key Insights and Ideas

  • Beyond Shallow Semantics: The paper argues that existing S2TT semantic communication systems extract only "shallow" semantic features tied to the source language. Ross-S2T aims to extract "deep" semantic features that span across languages, enabling more direct translation.
  • GANs for Semantic Recovery: Using GANs to estimate and recover lost semantic information in corrupted speech is a novel approach for this domain, allowing the system to "fill in the blanks" of degraded audio.
  • The Power of Probing: The "semantic impairment probe" is a clever mechanism to explicitly identify and target semantic errors. This targeted correction is more effective than a general compensation approach.
  • End-to-End Robustness: The system is designed for end-to-end training and robust performance, meaning it can handle corrupted inputs without requiring separate, pre-trained modules for noise reduction or translation.

Key Results

The paper demonstrates the superiority of Ross-S2T through simulation results, primarily measured by Semantic Textual Similarity (STS) scores.

  • Robustness to Noise: Under Rayleigh channels with a Signal-to-Noise Ratio (SNR) of 1 dB, Ross-S2T achieves an STS score of over 0.5, significantly outperforming a conventional speech transmission system (which scores below 0.4).
  • Comparison to DeepSC-S2T: The paper's earlier version, DeepSC-S2T, which is not designed for corrupted speech, shows a significantly inferior capability in recovering impaired semantic information compared to the proposed Ross-S2T. This highlights the effectiveness of the dual-compensator mechanism.

Limitations and Future Work

  • Limitations: The paper focuses on specific types of semantic impairments (background noise and external speech interference). The performance with other types of corruption or more severe degradation is not explicitly discussed. The reliance on accurate channel state information (CSI) for evaluation is also noted.
  • Future Work: Not explicitly discussed in the paper, but potential future directions could include exploring different types of semantic impairments, investigating real-time implementation challenges, and extending the framework to other speech-related tasks or multimodal semantic communication scenarios.

Summary updated

BibTeX
@inproceedings{weng2025robust,
  author = {Zhenzi Weng and Zhijin Qin and Geoffrey Ye Li},
  title = {{Robust Semantic Communications for Speech Transmission}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=nuKTUtCumT},
}
Figure