A Neural Vocoder With Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis

Yang Ai, Zhen-Hua Ling

TL;DR

HiNet is a novel neural vocoder that reconstructs high-quality speech by hierarchically predicting amplitude and phase spectra rather than generating waveforms sample-by-sample. By eliminating autoregressive structures, HiNet achieves speech naturalness comparable to state-of-the-art models like WaveRNN while running significantly faster—generating 1 second of audio in just 0.19 seconds on a GPU.

What Problem Does This Paper Address?

In speech synthesis, a "vocoder" is the component that converts abstract acoustic features (like pitch and spectral envelopes) into audible waveforms. Traditional vocoders (e.g., STRAIGHT) often sound "buzzy" or robotic because they lose fine spectral details and phase information.

Recent neural vocoders like WaveNet and WaveRNN produce incredibly natural speech, but they have a major drawback: they are autoregressive. They generate audio one sample at a time (e.g., 16,000 samples for one second of audio), making them computationally expensive and slow for real-time applications. While non-autoregressive alternatives like WaveGlow exist, they are often massive and difficult to train. HiNet seeks a "middle ground": a model that is easy to train, highly efficient through parallelization, and capable of producing high-fidelity speech.

Methods

The core innovation of HiNet is its hierarchical generation strategy, splitting the task into two specialized predictors:

  1. Amplitude Spectrum Predictor (ASP): A simple Deep Neural Network (DNN) that predicts the log amplitude spectra (LAS) from input acoustic features. To prevent the "over-smoothing" common in neural networks (which leads to muffled audio), the authors incorporate Generative Adversarial Networks (GANs) to ensure the predicted spectra match the distribution of natural speech.
  2. Phase Spectrum Predictor (PSP): Phase is notoriously difficult to model directly due to "phase warping." Instead of predicting phase values, the PSP uses a non-autoregressive Neural Source-Filter (NSF) waveform generator as an intermediate step. It generates a signal, and the phase is then extracted from that signal.
  3. Short-Time Fourier Synthesis: Finally, the outputs from the ASP and PSP are combined using traditional signal processing to reconstruct the final waveform.

Crucially, neither the ASP nor the PSP contains autoregressive connections, allowing the entire process to be parallelized on a GPU.

Datasets

The researchers evaluated HiNet using:

  • CMU-ARCTIC: A standard benchmark featuring speakers slt (female) and bdl (male).
  • Internal Chinese Dataset: A much larger dataset (19.5 hours) from a female speaker used to test how the model scales with more data.

Key Insights and Ideas

  • Phase via Proxy: The "aha moment" here is realizing that you don't need to predict phase directly. By using a lightweight neural waveform generator specifically to "suggest" a phase, the model bypasses the mathematical headaches of circular phase distributions.
  • Complexity vs. Quality: The authors discovered that the performance of HiNet is relatively insensitive to the complexity of the waveform generator in the PSP. This allowed them to simplify the model (creating "HiNet-S") to boost speed without a significant drop in audio quality.
  • The Importance of Initial Phase: A key technical finding was that pre-calculating the initial phase for voiced segments during training was essential. Without this "hint," the model struggled to learn phase information through waveform loss alone.
  • GANs for Detail: Using GANs specifically for the amplitude spectra helped recover high-frequency details (formants) that are often lost when using standard Mean Square Error (MSE) loss functions.

Key Results

  • Naturalness: In MUSHRA (subjective listening) tests, HiNet significantly outperformed the conventional STRAIGHT vocoder and a 16-bit WaveNet implementation. It achieved parity with the high-quality (but slower) 16-bit WaveRNN.
  • Efficiency: HiNet achieved a Real-Time Factor (RTF) of 0.19 on a GPU. For context, the WaveNet implementation used for comparison had an RTF of 222 (meaning it took 222 seconds to generate 1 second of audio).
  • Inference Speed: Even on a CPU, the simplified HiNet-S was nearly 80 times faster than WaveRNN.
  • Data Scaling: Tests on the large Chinese dataset showed that the GAN-based approach significantly improved spectral clarity as training data increased, bringing the spectrograms very close to natural speech.

Limitations and Future Work

  • Dependency on Acoustic Models: HiNet still requires high-quality acoustic features from an external model (like an HMM or another DNN) to function.
  • Data Requirements: While efficient, the GAN-based components require a decent amount of data to stabilize and avoid artifacts.
  • Future Directions: The authors suggest applying the HiNet architecture to other speech tasks, such as voice conversion and bandwidth extension, where efficient, high-quality waveform reconstruction is equally critical.

Summary updated

Abstract

This article presents a neural vocoder named HiNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra hierarchically. Different from existing neural vocoders such as WaveNet, SampleRNN and WaveRNN which directly generate waveform samples using single neural networks, the HiNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a simple DNN model which predicts log amplitude spectra (LAS) from acoustic features. The predicted LAS are sent into the PSP for phase recovery. Considering the issue of phase warping and the difficulty of phase modeling, the PSP is constructed by concatenating a neural source-filter (NSF) waveform generator with a phase extractor. We also introduce generative adversarial networks (GANs) into both ASP and PSP. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by short-time Fourier synthesis. Since there are no autoregressive structures in both predictors, the HiNet vocoder can generate speech waveforms with high efficiency. Objective and subjective experimental results show that our proposed HiNet vocoder achieves better naturalness of reconstructed speech than the conventional STRAIGHT vocoder, a 16-bit WaveNet vocoder using open source implementation and an NSF vocoder with similar complexity to the PSP and obtains similar performance with a 16-bit WaveRNN vocoder. We also find that the performance of HiNet is insensitive to the complexity of the neural waveform generator in PSP to some extend. After simplifying its model structure, the time consumed for generating 1 s waveforms of 16 kHz speech using a GPU can be further reduced from 0.34 s to 0.19 s without significant quality degradation.
BibTeX
@article{ai2020neural,
  author = {Yang Ai and Zhen-Hua Ling},
  title = {{A Neural Vocoder With Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=RFqrimcFED},
}

Loading related content...

Figure