Abstract
Recent progress in deep generative models has improved the quality of neural vocoders in the speech domain. However, generating a high-quality singing voice remains challenging due to a wider variety of musical expressions in pitch, loudness, and pronunciations. In this work, we propose a hierarchical diffusion model for singing voice neural vocoders. The proposed method consists of multiple diffusion models operating at different sampling rates; the model at the lowest sampling rate focuses on generating accurate low-frequency components such as pitch, and other models progressively generate the waveform at higher sampling rates based on the data at the lower sampling rate and acoustic features. Experimental results show that the proposed method produces high-quality singing voices for multiple singers, outperforming state-of-the-art neural vocoders with a similar range of computational costs.
Introduction
Neural vocoders, which synthesize audible waveforms from abstract acoustic features like mel-spectrograms, have reached high levels of performance for speech. However, the singing voice presents a significantly more complex challenge. Singing involves a broader range of pitch, dynamic loudness, and intricate musical expressions such as vibrato and falsetto. Standard models often struggle due to data scarcity in high-quality singing datasets and the inherent trade-off between capturing long-term melodic consistency and fine-grained high-frequency textures.
To address these challenges, researchers from Sony introduced the Hierarchical PriorGrad (HPG) model. By nesting multiple diffusion models that operate at progressively higher sampling rates, HPG captures both broad melodic structures and detailed acoustic nuances more effectively than single-resolution models.
Proposed Method: Hierarchical PriorGrad (HPG)
Hierarchical Architecture
The core of HPG is a "cascade" of $N$ independent diffusion models, each operating at a different sampling rate (e.g., 6 kHz and 24 kHz). During training, each stage is optimized to denoise signals at its specific resolution. During inference, the process is progressive: the lowest-resolution model generates a base signal, and subsequent models in the chain are conditioned on both the input mel-spectrograms and the output of the preceding lower-resolution stage.
This hierarchy offers a significant "receptive field advantage." By using the same network architecture at lower sampling rates, the model's effective receptive field covers a much longer temporal window. This allows the low-resolution stages to focus entirely on long-term structures like pitch and vibrato, while high-resolution stages specialize in high-frequency textures.
Anti-Aliasing Filter
A critical technical challenge in hierarchical generation is the "gap" between training and inference. During training, higher stages are conditioned on ground-truth downsampled audio. During inference, they are conditioned on generated audio from the previous stage, which may contain artifacts. Specifically, without intervention, noisy artifacts often appear at the Nyquist frequencies of the lower stages. To mitigate this, an anti-aliasing filter is employed during the inference phase.
Computational Efficiency
To maintain a Real-Time Factor (RTF) comparable to standard models, the authors reduced the number of layers in each hierarchical stage (e.g., $L=24$ instead of the standard $L=30$). This optimization allows a 2-stage HPG model to achieve superior quality without a significant increase in computational overhead compared to a single-stage PriorGrad model.
Experimental Evaluation and Results
The models were evaluated on singing voice datasets including NUS48E and NHSS. The proposed HPG-2 (2-stage) and HPG-3 (3-stage) models were compared against state-of-the-art baselines like Parallel WaveGAN (PWG) and PriorGrad.
- Subjective Quality (MOS): HPG-2 achieved a score of 3.95, significantly outperforming PriorGrad (3.60) and PWG (2.15).
- Preference: In head-to-head tests, raters preferred HPG-2 over PriorGrad 85.3% of the time.
- Objective Metrics: HPG-3 achieved the best scores in Pitch Mean Absolute Error (PMAE) and Voicing Decision Error (VDE), confirming its superior ability to track complex singing melodies and stable vibrato transitions.
Conclusion
The Hierarchical PriorGrad model effectively addresses the complexities of singing voice synthesis by decomposing the generation task across multiple resolutions. This approach improves melodic accuracy and audio crispness while maintaining competitive inference speeds. Future work may explore the applicability of this hierarchical diffusion framework to general music and environmental sound synthesis.
Summary updated