Introduction
Statistical Parametric Speech Synthesis (SPSS) has seen significant advancements with the adoption of Long Short-Term Memory (LSTM) networks. While LSTMs excel at capturing long-range temporal dependencies in speech, their complex architecture—comprising input, output, and forget gates—remains largely a "black box." This research aims to demystify these components to determine which are essential for high-quality speech generation and to optimize the architecture for efficiency.
Visualizing the Internal State
To understand how LSTMs model speech, the researchers performed a visual analysis of internal activations. A key discovery was the high degree of physical grounding in the network's memory. Specifically, certain cell states were found to track acoustic trajectories directly, showing that the internal "memory" is physically tracking the shape of the speech wave.
Furthermore, the forget gate was observed to act as a phoneme boundary detector. Its activations typically peak at the start of a phoneme and decay linearly, effectively "resetting" the memory for each new linguistic unit. This suggests that the forget gate is the primary mechanism for segmenting the continuous speech stream.
Ablation Studies and the Simplified LSTM (S-LSTM)
The researchers conducted ablation studies by systematically removing components: peephole connections (NPH), input gates (NIG), forget gates (NFG), and output gates (NOG). The results indicated that while removing the forget gate (NFG) severely degraded quality, the input and output gates were largely redundant for this task.
Based on these insights, a Simplified LSTM (S-LSTM) was proposed. The S-LSTM eliminates the input gate, output gate, and peepholes. It utilizes a single forget gate to balance the previous state and the current input, significantly streamlining the recurrent unit.
Performance and Efficiency
The S-LSTM achieves substantial improvements in computational efficiency without sacrificing speech quality. Quantitative analysis shows a 50% reduction in parameters and a significant boost in generation speed compared to the standard LSTM model.
Subjective preference tests confirmed that listeners could not significantly distinguish between speech generated by the full LSTM and the S-LSTM. In contrast, the NFG variant (no forget gate) was consistently rated as inferior, reinforcing the forget gate's status as the most vital component of the architecture.
Conclusion
By opening the LSTM "black box," this study demonstrates that much of the standard architecture's complexity is unnecessary for speech synthesis. The forget gate's role in boundary detection is paramount. The resulting S-LSTM offers a more efficient, faster alternative that maintains state-of-the-art quality, paving the way for more specialized and lightweight recurrent models in speech technology.
Summary updated