Diffusion-based Signal Refiner for Speech Separation

Masato Hirano, Kazuki Shimada, Yuichiro Koyama, Shusuke Takahashi, Yuki MitsufujiarXiv 2023 · arXiv · OpenReview

Abstract

Modern deep neural network (DNN) models for speech separation and enhancement have achieved impressive results in objective metrics like Signal-to-Distortion Ratio (SDR). However, these models often introduce perceptually unnatural artifacts, such as "musical noise" or missing harmonic structures. We present Diffiner, a diffusion-based signal refiner designed to bridge this perceptual gap. By leveraging a generative prior learned strictly from clean speech, Diffiner restores degraded components of initial separations. Our shared Denoising Diffusion Restoration Model (DDRM) framework allows for model-agnostic refinement, significantly improving reference-free perceptual quality across various preceding architectures. Experimental results demonstrate that Diffiner provides clearer harmonic structures and higher human preference scores compared to state-of-the-art discriminative models.

The Perceptual Gap in Speech Processing

While modern DNN-based Speech Enhancement (SE) and Speech Separation (SS) models are effective at isolating speakers, they often suffer from a "perceptual gap." Because these models are typically trained to minimize the mathematical distance from a ground-truth signal, they can be overly aggressive. In the process of suppressing noise, they frequently remove fine harmonic structures or introduce watery artifacts. The resulting audio may achieve a high SDR score but sounds processed and unnatural to human listeners.

Furthermore, most existing generative solutions to this problem require massive datasets of paired noisy/clean audio and must be retrained whenever the underlying SE/SS model changes. Diffiner addresses these issues by acting as a versatile, model-agnostic post-processor that requires only a clean speech prior.

Methodology: The Diffiner Framework

Diffiner treats speech refinement as a linear inverse problem solved via a diffusion process. The core of the system is a Denoising Diffusion Probabilistic Model (DDPM) trained exclusively on clean speech (using the WSJ0 dataset). This allows the model to learn the manifold of natural speech in the time-frequency (STFT) domain.

Shared DDRM Update

For multi-speaker scenarios, we propose a "shared model" approach. As shown in the architecture diagram below, the system analyzes the mixture signal and the separated outputs simultaneously. This ensures that the refined tracks remain faithful to the original mixture while improving individual clarity through a shared DDRM update block.

The Diffiner architecture for speech separation, featuring a shared DDRM update process to refine multiple sources simultaneously.

Key Technical Innovations

  • Spectral Space Updates: Refinement occurs in the spectral space using Singular Value Decomposition (SVD). For enhancement tasks, the model estimates noise variance to determine which signal components to preserve and which to regenerate.
  • The "Generative Wiener Filter": The authors demonstrate that Diffiner’s optimal coefficients at each diffusion step are mathematically equivalent to a Wiener filter. However, instead of merely suppressing noise, it replaces unreliable parts of the signal with plausible, high-fidelity speech components.
  • Blending Strategy: To balance faithfulness to the original recording with naturalness, a blending coefficient ($\xi$) allows users to mix the refined output with the original processed signal, optimizing for both SDR and perceptual quality.

Experimental Results

Diffiner was evaluated using the WSJ0, CHiME-3, and WSJ0-2mix datasets. The results consistently showed significant improvements in reference-free metrics across all tested preceding models.

Perceptual Quality Gains

As shown in the table below, Diffiner (DDRM in Diffiner) significantly outperforms both the raw source and baseline diffusion models like DiffWave, especially in low Signal-to-Noise Ratio (SNR) conditions. For instance, at an SNR of 5 dB, Diffiner achieves a perceptual score of 3.612, compared to just 1.335 for the source and 1.693 for DiffWave.

Comparison of perceptual quality scores across various Signal-to-Noise Ratios (SNR), demonstrating Diffiner's superior performance over baselines.

Human Subjective Testing

In MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) tests involving 12 expert listeners, refined signals were consistently rated higher than the outputs of state-of-the-art models like MP-SENet and Sepformer. Qualitative spectrogram analysis confirmed that Diffiner successfully restored clear harmonic structures in high-frequency regions that were previously erased by preceding models.

Conclusion and Future Directions

Diffiner provides a powerful, model-agnostic tool for enhancing the perceptual quality of separated speech. While it significantly improves naturalness and harmonic restoration, the iterative nature of diffusion models (requiring $T=200$ steps in this study) presents a challenge for real-time implementation. Future work will focus on reducing computational costs and evaluating performance on more complex, "in the wild" recordings involving heavy reverberation and unpredictable environmental noise.

Summary updated

Abstract

We have developed a diffusion-based speech refiner that improves the reference-free perceptual quality of the audio predicted by preceding single-channel speech separation models. Although modern deep neural network-based speech separation models have show high performance in reference-based metrics, they often produce perceptually unnatural artifacts. The recent advancements made to diffusion models motivated us to tackle this problem by restoring the degraded parts of initial separations with a generative approach. Utilizing the denoising diffusion restoration model (DDRM) as a basis, we propose a shared DDRM-based refiner that generates samples conditioned on the global information of preceding outputs from arbitrary speech separation models. We experimentally show that our refiner can provide a clearer harmonic structure of speech and improves the reference-free metric of perceptual quality for arbitrary preceding model architectures. Furthermore, we tune the variance of the measurement noise based on preceding outputs, which results in higher scores in both reference-free and reference-based metrics. The separation quality can also be further improved by blending the discriminative and generative outputs.
BibTeX
@misc{hirano2023diffusion,
  author = {Masato Hirano and Kazuki Shimada and Yuichiro Koyama and Shusuke Takahashi and Yuki Mitsufuji},
  title = {{Diffusion-based Signal Refiner for Speech Separation}},
  year = {2023},
  eprint = {2305.05857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.05857},
}
Figure