Abstract
Speech Enhancement (SE) aims to recover clean speech from noisy signals. While diffusion models offer high-quality generative capabilities, their application in SE has been limited by high computational costs and domain-specific challenges. This paper introduces SRTNet, a novel "enhance-and-refine" framework that operates directly in the time domain. By utilizing a deterministic module to provide an initial estimate and a stochastic diffusion module to refine the residual, SRTNet achieves state-of-the-art quality. Notably, it converges nearly five times faster than existing diffusion-based baselines and demonstrates robust generalization to unseen noise conditions.
Introduction
The goal of Speech Enhancement is to improve the clarity of recorded signals by removing background noise. Generative models, such as GANs and VAEs, have seen success in this area, but Diffusion Models—despite their success in image generation—face significant hurdles in SE. These include high computational costs for training and sampling, phase distortions in frequency-domain methods, and the difficulty of generating high-fidelity speech from pure noise.
SRTNet addresses these issues by proposing a pure time-domain model that is efficient enough for practical use. It simplifies the generative task by focusing on the residual signal rather than the full speech distribution.
Methodology: The Enhance-and-Refine Paradigm
The core innovation of SRTNet is its two-stage architecture, which splits the enhancement task into deterministic and stochastic components.
Deterministic Module ($D_\theta$)
The first stage employs a standard neural network that processes noisy speech ($y$) to produce an initial, "rough" enhanced version ($y_{init}$). This provides a strong structural foundation for the subsequent refinement.
Stochastic Module ($S_\theta$)
The second stage is a conditional diffusion model. Unlike traditional diffusion models that attempt to model the entire speech distribution, $S_\theta$ learns to model the residual—the difference between the clean speech and the initial estimate. By focusing on this residual, the model faces a much simpler distributional task, leading to significantly faster convergence.
Technical Implementation
- Pure Time-Domain Processing: SRTNet operates directly on raw waveforms, avoiding the "phase problem" and extra processing steps associated with Short-Time Fourier Transforms (STFT).
- Continuous Noise Level Encoding: Instead of discrete time steps, the model uses continuous noise level encoding, allowing for more flexible sampling and faster inference.
- Joint Training: Both modules are trained together, ensuring that the stochastic module specifically learns to correct the errors produced by the deterministic module.
Experiments and Results
The performance of SRTNet was evaluated on the VoiceBank-DEMAND benchmark for matched conditions and a WSJ0 + MUSAN setup to test generalization to unseen noise and speakers.
Performance and Efficiency
SRTNet achieved a PESQ score of 2.69 on VoiceBank-DEMAND, outperforming previous diffusion models like CDiffuSE (2.52) and various GAN-based approaches.
As shown in the training curves, SRTNet reached convergence in just 3.6 days using two NVIDIA 3090 GPUs, whereas the baseline CDiffuSE required 17.75 days. This represents a nearly 5x improvement in training efficiency.
Generalization
In mismatched noise conditions, SRTNet demonstrated superior robustness compared to discriminative models like Demucs or Conv-TasNet. While discriminative models often overfit to specific training noise patterns, SRTNet's generative nature allows it to maintain a high PESQ of 2.56, whereas others saw significant performance drops.
Discussion and Conclusion
The "enhance-and-refine" paradigm proves that residuals are an "easy mode" for diffusion models in speech enhancement. By refining a mostly-correct signal, SRTNet achieves high fidelity with much lower computational overhead.
However, the model faces challenges in Low Signal-to-Noise Ratio (SNR) environments. If the initial noisy signal is extremely corrupted, the deterministic module may fail to provide a helpful starting point, making it difficult for the stochastic module to recover the clean speech. Future work will explore applying this paradigm in the time-frequency domain to better model phase and magnitude separately and improving robustness in extreme noise conditions.
Summary updated