Abstract
Self-supervised learning (SSL) representations, such as those from Wav2Vec2, have become a cornerstone for speech tasks like automatic speech recognition (ASR) and speaker identification. However, their utility in speech enhancement (SE) systems—especially under the constraints of on-device, real-time processing—is not yet well understood. This paper investigates the integration of SSL representations into a lightweight, causal SE framework. Through a systematic evaluation of feature concatenation, knowledge distillation, and pre-training techniques, we find that SSL embeddings offer surprisingly little value for the enhancement task in low signal-to-noise ratio (SNR) conditions. Our analysis suggests a fundamental gap between the linguistic information captured by SSL models and the qualitative acoustic features required for effective speech reconstruction.
Introduction
The rise of self-supervised learning has revolutionized speech processing. Models like Wav2Vec2 are often viewed as universal feature extractors, providing rich representations that simplify downstream tasks. While these models excel at understanding what is being said, their ability to help clean up how it sounds is less certain.
In on-device applications, such as mobile telephony or hearing aids, speech enhancement models must be causal (processing audio without looking into the future) and computationally efficient. Large SSL models are typically non-causal and computationally expensive, making them unsuitable for direct on-device deployment. This study explores whether the "wisdom" of these large models can be distilled into smaller, real-time SE models to improve performance in challenging, low-SNR environments.
Methodology
To evaluate the impact of SSL representations, we utilized a Gated Convolutional Recurrent Network (GCRN) as our baseline student model. The GCRN is a lightweight architecture optimized for real-time, causal speech enhancement. We explored three primary methods to incorporate knowledge from a pre-trained Wav2Vec2 teacher model:
- Feature Concatenation: Wav2Vec2 embeddings were directly fed into the SE model. While this violates on-device constraints (due to the size and non-causality of Wav2Vec2), it serves as a performance upper bound.
- Knowledge Distillation: We attempted to force the internal layers of the small GCRN model to mimic Wav2Vec2 embeddings. We tested several loss functions for this distillation, including L1 loss, Adversarial (GAN-based) loss, and Triplet loss.
- Pre-training: The SE model was initialized by training it to predict Wav2Vec2 embeddings from noisy speech, followed by fine-tuning on the actual enhancement task.
Experimental Setup
The study used the Deep Noise Suppression (DNS) Challenge corpus for training and validation, with clean speech and noise mixed at random SNRs between -5dB and 5dB. For testing, we used 500 samples from the Librispeech test-clean set, mixed with noise at a fixed, challenging -5dB SNR to simulate harsh real-world conditions.
Performance was measured using standard metrics: PESQ (Perceptual Evaluation of Speech Quality), STOI (Short-Time Objective Intelligibility), and SI-SDR (Scale-Invariant Signal-to-Distortion Ratio).
Results and Analysis
The quantitative results provided a significant reality check for the application of SSL in this domain:
- Marginal Improvements: The best-performing distillation method (Distillation Output) achieved a PESQ of 1.60, barely outperforming the baseline GCRN's 1.59.
- Pre-training Inefficiency: Models pre-trained to predict SSL embeddings actually performed worse than those trained from scratch, yielding an SI-SDR of 8.60 compared to the baseline's 9.1.
- High-Frequency Loss: A qualitative analysis of the reconstructed speech revealed that models relying on distilled embeddings often lost energy in higher frequency bands, resulting in a "muffled" audio quality despite maintaining intelligibility.
Discussion: Why SSL Struggles in Speech Enhancement
The researchers identified several reasons for these underwhelming results:
Phonetic vs. Qualitative Information
Wav2Vec2 is optimized to capture the phonetic and linguistic structure of speech. However, speech enhancement requires qualitative information—tonality, loudness, and fine-grained voice quality—which SSL models tend to ignore or discard during their training for ASR-related tasks.
The "Tiny Variation" Problem
Analysis of Wav2Vec2 embeddings showed they are highly correlated over time, matching the length of phonemes. The information necessary for enhancement is often hidden in minute magnitude variations between frames. Small, on-device models struggle to capture these intricate details from the embeddings, often resulting in a "noise-like" average representation that fails to reconstruct the original signal accurately.
Conclusion
While SSL models are transformative for speech understanding, they are not a "silver bullet" for on-device speech enhancement. The linguistic focus of models like Wav2Vec2 does not naturally translate to the acoustic reconstruction needs of SE. Future research must focus on developing SSL frameworks that preserve qualitative acoustic features or finding more sophisticated ways to bridge the gap between linguistic understanding and signal reconstruction for real-time, resource-constrained environments.
Summary updated