Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution

Hao Tang, Chin-Yun Yu, Sung-Lin Yeh, György Fazekas, Hao Tang 0002ICASSP 2023 · DOI

TL;DR

This paper introduces a novel sampling method for diffusion models that significantly improves speech super-resolution (SR) by better communicating low-resolution audio information during the generation process. This approach acts as a drop-in replacement for existing methods and, when combined with unconditional diffusion models, achieves state-of-the-art results on a challenging benchmark.

What Problem Does This Paper Address?

Speech super-resolution (SR), also known as bandwidth extension, aims to reconstruct high-resolution speech from low-resolution audio. While diffusion models (DMs) have shown promise in this area, existing methods often struggle to perfectly reconstruct the high-frequency components, leading to deviations between the input and output. This is partly because the low-resolution audio is typically only used to condition the noise predictor, and the stochastic nature of the diffusion sampling process can introduce inaccuracies. The paper tackles this by proposing a new sampling strategy that more effectively leverages the low-resolution input throughout the entire reverse diffusion process.

Methods

The core contribution is a new conditional sampling algorithm for Variational Diffusion Models (VDMs) applied to speech SR. The authors frame SR as a frequency-domain inpainting problem. Their method modifies the reverse diffusion step, p(zt-1|zt), to incorporate the low-resolution audio y.

Specifically, they leverage the frequency-domain representation of the diffusion process. For frequencies where the low-resolution audio y is expected to contain information (i.e., below the downsampling cutoff frequency), their method directly uses the upsampled low-resolution audio ŷ to guide the sampling. For higher frequencies, where information is missing, they still rely on the noise predictor's estimate, but this estimate is implicitly guided by the low-frequency information that has been "inpainted."

This approach can be integrated with any existing diffusion model, whether it's trained unconditionally or conditioned on y. They also incorporate the Manifold Constrained Gradient (MCG) method to further refine the sampling process.

Datasets

The primary dataset used for training and evaluation is the VCTK-0.92 dataset. This dataset contains approximately 44 hours of speech from 108 English speakers with diverse accents. Experiments were conducted at both 48 kHz and 16 kHz sampling rates. The paper also mentions using two configurations for low-pass filtering during downsampling: a spectrogram-based method and a sinc filter method.

Key Insights and Ideas

  • Information Injection Throughout Sampling: The key insight is that by injecting the low-resolution audio information directly into the reverse sampling process, not just as a conditioning signal for the noise predictor, the model can more accurately reconstruct the missing high-frequency content.
  • Frequency-Domain Inpainting: Framing SR as a frequency-domain inpainting problem provides a principled way to leverage the known low-frequency information to guide the reconstruction of missing high frequencies.
  • Drop-in Replacement: The proposed sampling method is designed to be a drop-in replacement for existing sampling processes, making it easy to integrate and improve upon current diffusion-based SR models.
  • Generalization with Unconditional Models: Combining the new sampling method with an unconditional diffusion model (UDM) allows for generalization across different upscaling ratios and downsampling filter types without retraining the core diffusion model.
  • Robustness to Filters: The method demonstrates robustness to different zero-phase downsampling filters, a desirable trait for real-world applications.

Key Results

The paper reports state-of-the-art results on the 48 kHz VCTK Multi-Speaker benchmark.

  • Their proposed method, UDM+, achieved a reduction of 0.11 to 0.13 in Log-Spectral-Distance (LSD) compared to existing diffusion baselines like NU-Wave.
  • UDM+ outperformed baselines in most settings, particularly in terms of LSD.
  • For the 48 kHz experiment, the LSD-LF (low-frequency LSD) was significantly lower than reported for NU-Wave 2, indicating better retention of low frequencies.
  • The method showed minimal differences in LSD across different filter settings, highlighting its robustness.

Limitations and Future Work

  • Unconditional Generation Quality: The authors note that their UDMs performed poorly on unconditional generation. This suggests that the quality of the underlying unconditional diffusion model is crucial for the performance of their SR method, and there is room for improvement in training better unconditional models.
  • PESQ Score: While LSD scores were excellent, the perceptual metric PESQ was not as strong for the UDM+ model. The authors suggest that removing the MCG component might improve PESQ, but this would come at the cost of significantly worse LSD.
  • Future Work: The paper implies that improving the quality of the base unconditional diffusion models would further enhance the SR performance. Additionally, exploring different MCG step sizes or alternative regularization techniques could be avenues for future research.

Summary updated

BibTeX
@inproceedings{tang2023conditioning,
  author = {Hao Tang and Chin-Yun Yu and Sung-Lin Yeh and György Fazekas and Hao Tang},
  title = {{Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095103},
}
Figure