Hierarchical Diffusion Models for Singing Voice Neural Vocoder

Naoya Takahashi, Mayank Kumar Singh, Yuki Mitsufuji

TL;DR

Researchers from Sony have developed a Hierarchical Diffusion Model (HPG) specifically for singing voice synthesis. By nesting multiple diffusion models that operate at progressively higher sampling rates, the system captures both the broad melodic structure (pitch/vibrato) and fine-grained acoustic details more effectively than standard single-resolution models, outperforming state-of-the-art vocoders like PriorGrad and Parallel WaveGAN.

What Problem Does This Paper Address?

Neural vocoders—the components that turn abstract acoustic features into audible waveforms—have become highly effective for speech. However, singing voice synthesis remains a "boss level" challenge. Singing involves a much wider range of pitch, loudness, and complex musical expressions like vibrato and falsetto.

Existing models often struggle with these nuances because:

  1. Data Scarcity: High-quality singing datasets are much smaller than speech datasets.
  2. Resolution Trade-offs: A model focusing on high-frequency details might miss the long-term consistency needed for a steady pitch, while a model focusing on pitch might lack "crispness."
  3. Inference Speed: Diffusion models (DDPMs) produce high-quality audio but are traditionally slow, requiring many iterations to "denoise" a signal into clear audio.

Methods

The authors propose a hierarchical approach built upon PriorGrad, an optimized diffusion model.

The Hierarchical Architecture

Instead of one model trying to generate a 24kHz signal all at once, the authors use a "cascade" of models:

  • Multi-Resolution Training: They train $N$ independent diffusion models at different sampling rates (e.g., 6kHz and 24kHz).
  • Progressive Conditioning: During inference, the lowest-resolution model generates a base signal. The next model up the chain is conditioned on both the acoustic features (mel-spectrograms) and the output of the lower-resolution model.
  • Anti-Aliasing Filter: A key technical contribution is the use of an anti-aliasing filter during inference. Without it, the "gap" between training (using ground-truth downsampled audio) and inference (using generated audio) creates noisy artifacts at the Nyquist frequencies of the lower stages.

Efficiency Tweaks

To keep the computational cost (Real-Time Factor) comparable to standard models, they reduced the number of layers in each hierarchical stage ($L=24$ instead of the standard $L=30$). This allows a 2-stage model to run at roughly the same speed as a single-stage PriorGrad model.

Datasets

The models were trained and evaluated on three singing voice datasets:

  • NUS48E: 12 singers, 4 songs each.
  • NHSS: 10 singers, 10 songs each.
  • Internal Corpus: 8 singers, 50–80 songs each. All data was resampled to 24kHz for consistency.

Key Insights and Ideas

  • Receptive Field Advantage: By using the same network architecture at lower sampling rates, the model's "view" (receptive field) covers a much longer period of time. This allows the low-res model to focus entirely on long-term structures like pitch and vibrato.
  • Frequency Specialization: Ablation studies showed that the model naturally learns a "division of labor." The high-res stages ignore the low-frequency acoustic features and instead "copy" the structure from the low-res stage, focusing their own power on generating high-frequency textures.
  • Solving the "Vibrato Shake": Standard GAN-based vocoders (like Parallel WaveGAN) often produce "unnatural shakes" in pitch during vibrato. The diffusion-based hierarchical approach provides much more stable and natural pitch transitions.

Key Results

The proposed HPG-2 (2-stage) and HPG-3 (3-stage) models were compared against Parallel WaveGAN (PWG) and PriorGrad.

  • Subjective Quality (MOS): On a 5-point scale, HPG-2 achieved a 3.95, significantly higher than PriorGrad (3.60) and PWG (2.15).
  • Preference: In head-to-head listening tests, raters preferred HPG-2 over PriorGrad 85.3% of the time.
  • Objective Metrics: HPG-3 achieved the best scores in Pitch Mean Absolute Error (PMAE) and Voicing Decision Error (VDE), confirming its superior ability to track complex singing melodies.
  • Scaling: The authors found that simply making a standard PriorGrad model larger (PriorGrad-L) didn't improve quality as much as adding a hierarchical stage did, proving that the architecture—not just the parameter count—is the secret sauce.

Limitations and Future Work

While the results for singing are impressive, the authors note that this specific study was limited to that domain. They suggest that future work should explore whether this hierarchical diffusion approach provides similar benefits for:

  • Standard speech synthesis.
  • General music generation.
  • Environmental sound synthesis.

Additionally, while HPG-2 is efficient, HPG-3 introduces a ~30% increase in computational cost, suggesting a diminishing return for adding too many levels to the hierarchy.

Summary updated

Abstract

Recent progress in deep generative models has improved the quality of neural vocoders in speech domain. However, generating a high-quality singing voice remains challenging due to a wider variety of musical expressions in pitch, loudness, and pronunciations. In this work, we propose a hierarchical diffusion model for singing voice neural vocoders. The proposed method consists of multiple diffusion models operating in different sampling rates; the model at the lowest sampling rate focuses on generating accurate low-frequency components such as pitch, and other models progressively generate the waveform at higher sampling rates on the basis of the data at the lower sampling rate and acoustic features. Experimental results show that the proposed method produces high-quality singing voices for multiple singers, outperforming state-of-the-art neural vocoders with a similar range of computational costs.
BibTeX
@inproceedings{takahashi2023hierarchical,
  author = {Naoya Takahashi and Mayank Kumar Singh and Yuki Mitsufuji},
  title = {{Hierarchical Diffusion Models for Singing Voice Neural Vocoder}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095749},
}

Loading related content...

Figure