TL;DR
HiNet is a novel neural vocoder that reconstructs high-quality speech by hierarchically predicting amplitude and phase spectra rather than generating waveforms sample-by-sample. By eliminating autoregressive structures, HiNet achieves speech naturalness comparable to state-of-the-art models like WaveRNN while running significantly faster—generating 1 second of audio in just 0.19 seconds on a GPU.
What Problem Does This Paper Address?
In speech synthesis, a "vocoder" is the component that converts abstract acoustic features (like pitch and spectral envelopes) into audible waveforms. Traditional vocoders (e.g., STRAIGHT) often sound "buzzy" or robotic because they lose fine spectral details and phase information.
Recent neural vocoders like WaveNet and WaveRNN produce incredibly natural speech, but they have a major drawback: they are autoregressive. They generate audio one sample at a time (e.g., 16,000 samples for one second of audio), making them computationally expensive and slow for real-time applications. While non-autoregressive alternatives like WaveGlow exist, they are often massive and difficult to train. HiNet seeks a "middle ground": a model that is easy to train, highly efficient through parallelization, and capable of producing high-fidelity speech.
Methods
The core innovation of HiNet is its hierarchical generation strategy, splitting the task into two specialized predictors:
- Amplitude Spectrum Predictor (ASP): A simple Deep Neural Network (DNN) that predicts the log amplitude spectra (LAS) from input acoustic features. To prevent the "over-smoothing" common in neural networks (which leads to muffled audio), the authors incorporate Generative Adversarial Networks (GANs) to ensure the predicted spectra match the distribution of natural speech.
- Phase Spectrum Predictor (PSP): Phase is notoriously difficult to model directly due to "phase warping." Instead of predicting phase values, the PSP uses a non-autoregressive Neural Source-Filter (NSF) waveform generator as an intermediate step. It generates a signal, and the phase is then extracted from that signal.
- Short-Time Fourier Synthesis: Finally, the outputs from the ASP and PSP are combined using traditional signal processing to reconstruct the final waveform.
Crucially, neither the ASP nor the PSP contains autoregressive connections, allowing the entire process to be parallelized on a GPU.
Datasets
The researchers evaluated HiNet using:
- CMU-ARCTIC: A standard benchmark featuring speakers slt (female) and bdl (male).
- Internal Chinese Dataset: A much larger dataset (19.5 hours) from a female speaker used to test how the model scales with more data.
Key Insights and Ideas
- Phase via Proxy: The "aha moment" here is realizing that you don't need to predict phase directly. By using a lightweight neural waveform generator specifically to "suggest" a phase, the model bypasses the mathematical headaches of circular phase distributions.
- Complexity vs. Quality: The authors discovered that the performance of HiNet is relatively insensitive to the complexity of the waveform generator in the PSP. This allowed them to simplify the model (creating "HiNet-S") to boost speed without a significant drop in audio quality.
- The Importance of Initial Phase: A key technical finding was that pre-calculating the initial phase for voiced segments during training was essential. Without this "hint," the model struggled to learn phase information through waveform loss alone.
- GANs for Detail: Using GANs specifically for the amplitude spectra helped recover high-frequency details (formants) that are often lost when using standard Mean Square Error (MSE) loss functions.
Key Results
- Naturalness: In MUSHRA (subjective listening) tests, HiNet significantly outperformed the conventional STRAIGHT vocoder and a 16-bit WaveNet implementation. It achieved parity with the high-quality (but slower) 16-bit WaveRNN.
- Efficiency: HiNet achieved a Real-Time Factor (RTF) of 0.19 on a GPU. For context, the WaveNet implementation used for comparison had an RTF of 222 (meaning it took 222 seconds to generate 1 second of audio).
- Inference Speed: Even on a CPU, the simplified HiNet-S was nearly 80 times faster than WaveRNN.
- Data Scaling: Tests on the large Chinese dataset showed that the GAN-based approach significantly improved spectral clarity as training data increased, bringing the spectrograms very close to natural speech.
Limitations and Future Work
- Dependency on Acoustic Models: HiNet still requires high-quality acoustic features from an external model (like an HMM or another DNN) to function.
- Data Requirements: While efficient, the GAN-based components require a decent amount of data to stabilize and avoid artifacts.
- Future Directions: The authors suggest applying the HiNet architecture to other speech tasks, such as voice conversion and bandwidth extension, where efficient, high-quality waveform reconstruction is equally critical.
Summary updated