Investigating gated recurrent networks for speech synthesis

Zhizheng Wu, Simon KingICASSP 2016 · OpenReview

Introduction

Statistical Parametric Speech Synthesis (SPSS) has seen significant advancements with the adoption of Long Short-Term Memory (LSTM) networks. While LSTMs excel at capturing long-range temporal dependencies in speech, their complex architecture—comprising input, output, and forget gates—remains largely a "black box." This research aims to demystify these components to determine which are essential for high-quality speech generation and to optimize the architecture for efficiency.

Visualizing the Internal State

To understand how LSTMs model speech, the researchers performed a visual analysis of internal activations. A key discovery was the high degree of physical grounding in the network's memory. Specifically, certain cell states were found to track acoustic trajectories directly, showing that the internal "memory" is physically tracking the shape of the speech wave.

The 120th cell state (red dashed) shows a strong correlation with the 1st Mel-Cepstral Coefficient (blue solid) trajectory.

Furthermore, the forget gate was observed to act as a phoneme boundary detector. Its activations typically peak at the start of a phoneme and decay linearly, effectively "resetting" the memory for each new linguistic unit. This suggests that the forget gate is the primary mechanism for segmenting the continuous speech stream.

Ablation Studies and the Simplified LSTM (S-LSTM)

The researchers conducted ablation studies by systematically removing components: peephole connections (NPH), input gates (NIG), forget gates (NFG), and output gates (NOG). The results indicated that while removing the forget gate (NFG) severely degraded quality, the input and output gates were largely redundant for this task.

Based on these insights, a Simplified LSTM (S-LSTM) was proposed. The S-LSTM eliminates the input gate, output gate, and peepholes. It utilizes a single forget gate to balance the previous state and the current input, significantly streamlining the recurrent unit.

Performance and Efficiency

The S-LSTM achieves substantial improvements in computational efficiency without sacrificing speech quality. Quantitative analysis shows a 50% reduction in parameters and a significant boost in generation speed compared to the standard LSTM model.

Objective metrics and efficiency comparison across LSTM variants. The S-LSTM (bottom row) achieves the lowest parameter count and fastest generation time.

Subjective preference tests confirmed that listeners could not significantly distinguish between speech generated by the full LSTM and the S-LSTM. In contrast, the NFG variant (no forget gate) was consistently rated as inferior, reinforcing the forget gate's status as the most vital component of the architecture.

Conclusion

By opening the LSTM "black box," this study demonstrates that much of the standard architecture's complexity is unnecessary for speech synthesis. The forget gate's role in boundary detection is paramount. The resulting S-LSTM offers a more efficient, faster alternative that maintains state-of-the-art quality, paving the way for more specialized and lightweight recurrent models in speech technology.

Summary updated

Abstract

Recently, recurrent neural networks (RNNs) as powerful sequence models have re-emerged as a potential acoustic model for statistical parametric speech synthesis (SPSS). The long short-term memory (LSTM) architecture is particularly attractive because it addresses the vanishing gradient problem in standard RNNs, making them easier to train. Although recent studies have demonstrated that LSTMs can achieve significantly better performance on SPSS than deep feedforward neural networks, little is known about why. Here we attempt to answer two questions: a) why do LSTMs work well as a sequence model for SPSS; b) which component (e.g., input gate, output gate, forget gate) is most important. We present a visual analysis alongside a series of experiments, resulting in a proposal for a simplified architecture. The simplified architecture has significantly fewer parameters than an LSTM, thus reducing generation complexity considerably without degrading quality.
BibTeX
@inproceedings{wu2016investigating,
  author = {Zhizheng Wu and Simon King},
  title = {{Investigating gated recurrent networks for speech synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2016},
  url = {https://openreview.net/forum?id=VMVFECx6S61},
}
Figure