SELM: Speech Enhancement Using Discrete Tokens and Language Models

Ziqian Wang, Xinfa Zhu, Zihan Zhang, YuanJun Lv, Ning Jiang, Guoqing Zhao, Lei XieICASSP 2023

Abstract

Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception.

Introduction

Speech Enhancement (SE) has traditionally relied on discriminative models designed to minimize the distance between noisy inputs and clean targets. While effective, these models often struggle with generalization and can introduce audible distortions like "musical noise." Recently, generative models have emerged as a way to produce more natural-sounding speech.

SELM (Speech Enhancement using Language Models) introduces a new approach by treating noise reduction as a sequence-to-sequence translation task between discrete audio tokens. Unlike previous LM-based systems that require text transcripts, SELM uses self-supervised acoustic representations to bridge the gap between continuous audio and the semantic modeling power of Transformers.

Methodology

The SELM framework operates through a three-stage pipeline designed to move from raw audio to a semantic latent space and back to a clean waveform.

The SELM framework overview showing the three stages: Encoding, Modeling, and Decoding.
  1. Encoding (Waveform to Tokens): Raw noisy waveforms are processed by a pre-trained WavLM model. These continuous features are then discretized via k-means clustering, transforming the audio into a sequence of discrete "acoustic tokens" that capture both acoustic and semantic information.
  2. Modeling (Speech LM): At the core of SELM is a decoder-only Transformer architecture. It is trained to predict clean token sequences from noisy ones. By using a bidirectional attention mechanism on the noisy input, the model leverages full contextual information to "fill in" speech details obscured by noise.
  3. Decoding (Tokens to Waveform): To reconstruct the audio, a De-tokenizer (Conformer-based) maps the tokens back into continuous WavLM features. A HiFi-GAN vocoder then synthesizes the final enhanced speech signal.

Experimental Setup

The model was pre-trained on a 2,000-hour subset of WeNetSpeech and GigaSpeech. The Speech LM was specifically trained using LibriMix, Voice Bank (VCTK), and DNS Challenge datasets. Evaluation was conducted on the DNS Challenge 2021 test set and simulated noisy VCTK data to measure Word Error Rate (WER) and perceptual quality.

Results and Discussion

SELM demonstrates a significant advantage in intelligibility and subjective quality over traditional discriminative and other generative baselines.

Performance comparison of SELM against discriminative (D) and generative (G) baselines on WER, QMOS, and SMOS.
  • Superior Intelligibility: SELM achieved a Word Error Rate (WER) of 4.32%, outperforming the best discriminative baseline, Inter-SubNet (5.03%), and generative models like StoRM (6.14%).
  • Subjective Quality: In Mean Opinion Score tests, SELM reached a QMOS of 3.43, indicating that human listeners found the output more natural and pleasant than competing models.
  • The Token Trade-off: Experiments with the number of clusters ($K$) revealed that while fewer clusters ($K=300$) aid convergence and objective scores, larger vocabularies ($K=4096$) allow for more nuanced modeling of pitch and emotion.

While SELM excels in perceptual coherence, it does not focus on sample-by-sample waveform matching, which can lead to lower scores on metrics like Si-SNR despite sounding better to human ears. Future work will explore extending this LM-based approach to tasks like dereverberation and speech separation.

Conclusion

SELM represents a shift in speech enhancement from signal-level processing to semantic-level modeling. By leveraging the contextual power of Transformers and discrete tokens, it provides a robust solution for generating high-quality, intelligible speech in noisy environments without the need for text guidance.

Summary updated

Abstract

Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception. Our demos are available https://honee-w.github.io/SELM/.
BibTeX
@inproceedings{wang2023selm,
  author = {Ziqian Wang and Xinfa Zhu and Zihan Zhang and YuanJun Lv and Ning Jiang and Guoqing Zhao and Lei Xie},
  title = {{SELM: Speech Enhancement Using Discrete Tokens and Language Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
}
Figure