Hierarchical Diffusion Models for Singing Voice Neural Vocoder

Naoya Takahashi, Mayank Kumar Singh, Yuki Mitsufuji

Abstract

Recent progress in deep generative models has improved the quality of neural vocoders in the speech domain. However, generating a high-quality singing voice remains challenging due to a wider variety of musical expressions in pitch, loudness, and pronunciations. In this work, we propose a hierarchical diffusion model for singing voice neural vocoders. The proposed method consists of multiple diffusion models operating at different sampling rates; the model at the lowest sampling rate focuses on generating accurate low-frequency components such as pitch, and other models progressively generate the waveform at higher sampling rates based on the data at the lower sampling rate and acoustic features. Experimental results show that the proposed method produces high-quality singing voices for multiple singers, outperforming state-of-the-art neural vocoders with a similar range of computational costs.

Introduction

Neural vocoders, which synthesize audible waveforms from abstract acoustic features like mel-spectrograms, have reached high levels of performance for speech. However, the singing voice presents a significantly more complex challenge. Singing involves a broader range of pitch, dynamic loudness, and intricate musical expressions such as vibrato and falsetto. Standard models often struggle due to data scarcity in high-quality singing datasets and the inherent trade-off between capturing long-term melodic consistency and fine-grained high-frequency textures.

To address these challenges, researchers from Sony introduced the Hierarchical PriorGrad (HPG) model. By nesting multiple diffusion models that operate at progressively higher sampling rates, HPG captures both broad melodic structures and detailed acoustic nuances more effectively than single-resolution models.

Proposed Method: Hierarchical PriorGrad (HPG)

Hierarchical Architecture

The core of HPG is a "cascade" of $N$ independent diffusion models, each operating at a different sampling rate (e.g., 6 kHz and 24 kHz). During training, each stage is optimized to denoise signals at its specific resolution. During inference, the process is progressive: the lowest-resolution model generates a base signal, and subsequent models in the chain are conditioned on both the input mel-spectrograms and the output of the preceding lower-resolution stage.

Overview of the hierarchical diffusion process, illustrating the forward and reverse processes across different sampling rate stages.

This hierarchy offers a significant "receptive field advantage." By using the same network architecture at lower sampling rates, the model's effective receptive field covers a much longer temporal window. This allows the low-resolution stages to focus entirely on long-term structures like pitch and vibrato, while high-resolution stages specialize in high-frequency textures.

Anti-Aliasing Filter

A critical technical challenge in hierarchical generation is the "gap" between training and inference. During training, higher stages are conditioned on ground-truth downsampled audio. During inference, they are conditioned on generated audio from the previous stage, which may contain artifacts. Specifically, without intervention, noisy artifacts often appear at the Nyquist frequencies of the lower stages. To mitigate this, an anti-aliasing filter is employed during the inference phase.

Spectrograms of the noise predictor output without (a) and with (b) the anti-aliasing filter. The filter successfully removes the horizontal artifact at the Nyquist frequency.

Computational Efficiency

To maintain a Real-Time Factor (RTF) comparable to standard models, the authors reduced the number of layers in each hierarchical stage (e.g., $L=24$ instead of the standard $L=30$). This optimization allows a 2-stage HPG model to achieve superior quality without a significant increase in computational overhead compared to a single-stage PriorGrad model.

Experimental Evaluation and Results

The models were evaluated on singing voice datasets including NUS48E and NHSS. The proposed HPG-2 (2-stage) and HPG-3 (3-stage) models were compared against state-of-the-art baselines like Parallel WaveGAN (PWG) and PriorGrad.

  • Subjective Quality (MOS): HPG-2 achieved a score of 3.95, significantly outperforming PriorGrad (3.60) and PWG (2.15).
  • Preference: In head-to-head tests, raters preferred HPG-2 over PriorGrad 85.3% of the time.
  • Objective Metrics: HPG-3 achieved the best scores in Pitch Mean Absolute Error (PMAE) and Voicing Decision Error (VDE), confirming its superior ability to track complex singing melodies and stable vibrato transitions.

Conclusion

The Hierarchical PriorGrad model effectively addresses the complexities of singing voice synthesis by decomposing the generation task across multiple resolutions. This approach improves melodic accuracy and audio crispness while maintaining competitive inference speeds. Future work may explore the applicability of this hierarchical diffusion framework to general music and environmental sound synthesis.

Summary updated

Abstract

Recent progress in deep generative models has improved the quality of neural vocoders in speech domain. However, generating a high-quality singing voice remains challenging due to a wider variety of musical expressions in pitch, loudness, and pronunciations. In this work, we propose a hierarchical diffusion model for singing voice neural vocoders. The proposed method consists of multiple diffusion models operating in different sampling rates; the model at the lowest sampling rate focuses on generating accurate low-frequency components such as pitch, and other models progressively generate the waveform at higher sampling rates on the basis of the data at the lower sampling rate and acoustic features. Experimental results show that the proposed method produces high-quality singing voices for multiple singers, outperforming state-of-the-art neural vocoders with a similar range of computational costs.
BibTeX
@inproceedings{takahashi2023hierarchical,
  author = {Naoya Takahashi and Mayank Kumar Singh and Yuki Mitsufuji},
  title = {{Hierarchical Diffusion Models for Singing Voice Neural Vocoder}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095749},
}
Figure