A Neural Vocoder with Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis

Yang Ai, Zhen-Hua Ling

Abstract

This paper presents a neural vocoder named HiNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra hierarchically. Different from existing neural vocoders such as WaveNet and WaveRNN which directly generate waveform samples using single neural networks, the HiNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a simple DNN model which predicts log amplitude spectra (LAS) from acoustic features. The predicted LAS are sent into the PSP for phase recovery. Considering the issue of phase warping and the difficulty of phase modeling, the PSP is constructed by concatenating a neural source-filter (NSF) waveform generator with a phase extractor. We also introduce generative adversarial networks (GANs) into both ASP and PSP. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by short-time Fourier synthesis. Since there are no autoregressive structures in both predictors, the HiNet vocoder can generate speech waveforms with high efficiency. Objective and subjective experimental results show that our proposed HiNet vocoder achieves better naturalness of reconstructed speech than the conventional STRAIGHT vocoder and obtains similar performance with a 16-bit WaveRNN vocoder.

Introduction

In statistical parametric speech synthesis (SPSS), the vocoder is the critical component that converts abstract acoustic features into audible waveforms. Traditional vocoders like STRAIGHT often suffer from a "buzzy" or robotic quality due to the loss of fine spectral details and phase information. While recent neural vocoders like WaveNet and WaveRNN produce highly natural speech, they are typically autoregressive, generating audio one sample at a time. This sample-by-sample generation is computationally expensive and slow, often precluding real-time applications. HiNet addresses these challenges by proposing a non-autoregressive, hierarchical generation framework that is both efficient and capable of high-fidelity reconstruction.

Methods

The core innovation of HiNet is its hierarchical architecture, which decomposes waveform generation into two specialized, parallelizable tasks: amplitude prediction and phase recovery.

The overall framework of the HiNet vocoder, illustrating the hierarchical generation of amplitude and phase spectra.
  1. Amplitude Spectrum Predictor (ASP): A Deep Neural Network (DNN) predicts log amplitude spectra (LAS) from input acoustic features. To combat the "over-smoothing" effect common in neural network predictions—which often results in muffled audio—Generative Adversarial Networks (GANs) are integrated to ensure the predicted spectra match the distribution of natural speech.
  2. Phase Spectrum Predictor (PSP): Direct phase modeling is difficult due to phase warping. HiNet bypasses this by using a non-autoregressive Neural Source-Filter (NSF) waveform generator as an intermediate step. The PSP generates a signal from which the phase is then extracted.
  3. Synthesis: The predicted amplitude and extracted phase are combined using short-time Fourier synthesis to produce the final waveform. Because neither predictor is autoregressive, the entire inference process can be fully parallelized on a GPU.

Results and Discussion

The researchers evaluated HiNet using the CMU-ARCTIC benchmark and a large-scale internal Chinese dataset. Subjective listening tests using the MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) methodology were conducted to assess speech naturalness.

Subjective MUSHRA scores comparing HiNet against STRAIGHT, WaveNet, and WaveRNN baselines.

As shown in Figure 5, HiNet significantly outperformed the conventional STRAIGHT vocoder and a 16-bit WaveNet implementation. Most notably, it achieved parity with the high-quality 16-bit WaveRNN.

In terms of efficiency, HiNet demonstrated a massive speed advantage. It achieved a Real-Time Factor (RTF) of 0.19 on a GPU, meaning it can generate one second of audio in less than 0.2 seconds. In comparison, the WaveNet implementation used for benchmarking had an RTF of 222. Further simplification of the model (HiNet-S) allowed for even faster inference without significant degradation in audio quality, making it nearly 80 times faster than WaveRNN on a CPU.

Conclusion

HiNet provides a high-performance alternative to autoregressive neural vocoders. By leveraging a hierarchical approach to spectral generation and incorporating GANs for detail recovery, it achieves state-of-the-art naturalness while maintaining the efficiency required for real-time synthesis. Future work will explore the application of this architecture to related tasks such as voice conversion and bandwidth extension.

Summary updated

Abstract

This paper presents a neural vocoder named HiNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra hierarchically. Different from existing neural vocoders such as WaveNet, SampleRNN and WaveRNN which directly generate waveform samples using single neural networks, the HiNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a simple DNN model which predicts log amplitude spectra (LAS) from acoustic features. The predicted LAS are sent into the PSP for phase recovery. Considering the issue of phase warping and the difficulty of phase modeling, the PSP is constructed by concatenating a neural source-filter (NSF) waveform generator with a phase extractor. We also introduce generative adversarial networks (GANs) into both ASP and PSP. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by short-time Fourier synthesis. Since there are no autoregressive structures in both predictors, the HiNet vocoder can generate speech waveforms with high efficiency. Objective and subjective experimental results show that our proposed HiNet vocoder achieves better naturalness of reconstructed speech than the conventional STRAIGHT vocoder, a 16-bit WaveNet vocoder using open source implementation and an NSF vocoder with similar complexity to the PSP and obtains similar performance with a 16-bit WaveRNN vocoder. We also find that the performance of HiNet is insensitive to the complexity of the neural waveform generator in PSP to some extend. After simplifying its model structure, the time consumed for generating 1s waveforms of 16kHz speech using a GPU can be further reduced from 0.34s to 0.19s without significant quality degradation.
BibTeX
@article{ai2020neural,
  author = {Yang Ai and Zhen-Hua Ling},
  title = {{A Neural Vocoder with Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=RFqrimcFED},
}
Figure