Abstract
Modern deep neural network (DNN) models for speech separation and enhancement have achieved impressive results in objective metrics like Signal-to-Distortion Ratio (SDR). However, these models often introduce perceptually unnatural artifacts, such as "musical noise" or missing harmonic structures. We present Diffiner, a diffusion-based signal refiner designed to bridge this perceptual gap. By leveraging a generative prior learned strictly from clean speech, Diffiner restores degraded components of initial separations. Our shared Denoising Diffusion Restoration Model (DDRM) framework allows for model-agnostic refinement, significantly improving reference-free perceptual quality across various preceding architectures. Experimental results demonstrate that Diffiner provides clearer harmonic structures and higher human preference scores compared to state-of-the-art discriminative models.
The Perceptual Gap in Speech Processing
While modern DNN-based Speech Enhancement (SE) and Speech Separation (SS) models are effective at isolating speakers, they often suffer from a "perceptual gap." Because these models are typically trained to minimize the mathematical distance from a ground-truth signal, they can be overly aggressive. In the process of suppressing noise, they frequently remove fine harmonic structures or introduce watery artifacts. The resulting audio may achieve a high SDR score but sounds processed and unnatural to human listeners.
Furthermore, most existing generative solutions to this problem require massive datasets of paired noisy/clean audio and must be retrained whenever the underlying SE/SS model changes. Diffiner addresses these issues by acting as a versatile, model-agnostic post-processor that requires only a clean speech prior.
Methodology: The Diffiner Framework
Diffiner treats speech refinement as a linear inverse problem solved via a diffusion process. The core of the system is a Denoising Diffusion Probabilistic Model (DDPM) trained exclusively on clean speech (using the WSJ0 dataset). This allows the model to learn the manifold of natural speech in the time-frequency (STFT) domain.
Shared DDRM Update
For multi-speaker scenarios, we propose a "shared model" approach. As shown in the architecture diagram below, the system analyzes the mixture signal and the separated outputs simultaneously. This ensures that the refined tracks remain faithful to the original mixture while improving individual clarity through a shared DDRM update block.
Key Technical Innovations
- Spectral Space Updates: Refinement occurs in the spectral space using Singular Value Decomposition (SVD). For enhancement tasks, the model estimates noise variance to determine which signal components to preserve and which to regenerate.
- The "Generative Wiener Filter": The authors demonstrate that Diffiner’s optimal coefficients at each diffusion step are mathematically equivalent to a Wiener filter. However, instead of merely suppressing noise, it replaces unreliable parts of the signal with plausible, high-fidelity speech components.
- Blending Strategy: To balance faithfulness to the original recording with naturalness, a blending coefficient ($\xi$) allows users to mix the refined output with the original processed signal, optimizing for both SDR and perceptual quality.
Experimental Results
Diffiner was evaluated using the WSJ0, CHiME-3, and WSJ0-2mix datasets. The results consistently showed significant improvements in reference-free metrics across all tested preceding models.
Perceptual Quality Gains
As shown in the table below, Diffiner (DDRM in Diffiner) significantly outperforms both the raw source and baseline diffusion models like DiffWave, especially in low Signal-to-Noise Ratio (SNR) conditions. For instance, at an SNR of 5 dB, Diffiner achieves a perceptual score of 3.612, compared to just 1.335 for the source and 1.693 for DiffWave.
Human Subjective Testing
In MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) tests involving 12 expert listeners, refined signals were consistently rated higher than the outputs of state-of-the-art models like MP-SENet and Sepformer. Qualitative spectrogram analysis confirmed that Diffiner successfully restored clear harmonic structures in high-frequency regions that were previously erased by preceding models.
Conclusion and Future Directions
Diffiner provides a powerful, model-agnostic tool for enhancing the perceptual quality of separated speech. While it significantly improves naturalness and harmonic restoration, the iterative nature of diffusion models (requiring $T=200$ steps in this study) presents a challenge for real-time implementation. Future work will focus on reducing computational costs and evaluating performance on more complex, "in the wild" recordings involving heavy reverberation and unpredictable environmental noise.
Summary updated