Introduction
In the domain of Statistical Parametric Speech Synthesis (SPSS), the standard approach involves training neural networks to map linguistic features—such as phonemes, part-of-speech tags, and emphasis markers—to acoustic parameters like pitch ($F_0$) and spectral envelopes. A persistent challenge in this pipeline is the temporal resolution mismatch. While acoustic frames are typically generated at a high frequency (e.g., every 5 milliseconds), linguistic features exist at much coarser and varied timescales, such as phones, syllables, and words.
Traditionally, researchers have bypassed this issue by "flattening" and "upsampling" linguistic data to match the acoustic frame rate. However, this method introduces two significant drawbacks:
- Computational Redundancy: Repeating a word-level feature hundreds of times creates massive, highly redundant input vectors, leading to inefficient processing.
- Information Dilution: High-level linguistic structures, which are crucial for natural-sounding intonation and rhythm (prosody), become diluted when spread across numerous frames. This makes it difficult for standard models to capture the long-range dependencies necessary for expressive speech.
This paper introduces a Hierarchical Encoder-Decoder (HED) architecture designed to process linguistic features at their natural timescales, thereby improving both efficiency and the naturalness of synthetic speech.
The Hierarchical Encoder-Decoder (HED) Architecture
The core innovation of the HED model is an architecture that mirrors the inherent hierarchy of human language. Instead of pre-processing data into a flat sequence, the model handles features at multiple levels.
Key Components
- Hierarchical Encoder: The encoder is structured into distinct layers corresponding to words, syllables, and phones. Features are injected only at their relevant level. For instance, 300-dimensional GloVe word embeddings enter at the word layer, providing semantic context that informs the entire sentence's prosody.
- Progressive Upsampling: Rather than upsampling at the input stage, the HED model performs upsampling between its internal layers. Word-level hidden states are passed to the syllable level, which then informs the phone level. This ensures that the model "understands" the relationship between these units by design.
- Recurrent Decoder: The final phonetic encoding is processed by a recurrent decoder using SLSTM units. This decoder upsamples the phonetic information to the final acoustic frame rate, generating the parameters required for speech synthesis.
Experimental Evaluation
The authors evaluated the HED model using the 2016 Blizzard Challenge dataset, which contains approximately 4.33 hours of high-quality British English audiobook recordings. Audiobooks provide a rigorous test for speech synthesis because they require varied and expressive prosody, moving beyond the monotone delivery of standard virtual assistants.
Objective Results
The HED model demonstrated a clear advantage in modeling fundamental frequency ($F_0$), which is the primary acoustic correlate of pitch and intonation.
As shown in the results table, the HED-LSTM achieved a lower Root Mean Square Error (RMSE) for $F_0$ (50.20 Hz vs. 51.85 Hz) and a higher Pearson correlation compared to the baseline. Interestingly, while intonation improved, the Mel-cepstral distortion (MCD)—a measure of spectral quality—showed a very slight increase, suggesting a minor trade-off between prosodic accuracy and spectral texture.
Subjective Performance and Efficiency
Subjective listening tests confirmed these findings. In "blind" preference tests, human listeners preferred the HED model's output as more natural in 45% of cases, compared to 35% for the baseline Frame-LSTM.
Beyond quality, the HED architecture proved more computationally efficient. By moving upsampling inside the model and reducing the input dimensionality for lower layers, the HED-LSTM was approximately 18% faster at generating speech, reducing the generation time for 387 test utterances from 1910 seconds to 1566 seconds.
Conclusion and Future Work
The Hierarchical Encoder-Decoder model represents a significant step toward more linguistically-aware speech synthesis. By aligning the neural network's architecture with the multi-scale nature of language, the authors achieved superior prosody and improved generation speed.
Future research directions include integrating duration prediction directly into the hierarchical structure—removing the reliance on external HMM-based aligners—and exploring the use of direct waveform generators like WaveNet as the decoder to further enhance audio fidelity.
Summary updated