Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition

Xiangang Li, Xihong Wu

Abstract

Long short-term memory (LSTM) based acoustic modeling methods have recently been shown to give state-of-the-art performance on some speech recognition tasks. To achieve a further performance improvement, in this research, deep extensions on LSTM are investigated considering that deep hierarchical model has turned out to be more efficient than a shallow one. Motivated by previous research on constructing deep recurrent neural networks (RNNs), alternative deep LSTM architectures are proposed and empirically evaluated on a large vocabulary conversational telephone speech recognition task. Meanwhile, regarding to multi-GPU devices, the training process for LSTM networks is introduced and discussed. Experimental results demonstrate that the deep LSTM networks benefit from the depth and yield the state-of-the-art performance on this task.

Introduction

Context-dependent Deep Neural Network Hidden Markov Models (CD-DNN-HMMs) have long been the dominant framework for acoustic modeling in speech recognition. However, the inherently dynamic and sequential nature of speech suggests that Recurrent Neural Networks (RNNs)—specifically LSTMs—are theoretically better suited for the task. While LSTMs are "deep in time" due to their recurrence, standard single-layer LSTMs often lack the "spatial depth" or hierarchical representation that makes feed-forward Deep Neural Networks (DNNs) so powerful. This paper explores architectural strategies to add spatial depth to LSTMs, aiming to capture more complex acoustic representations for large vocabulary speech recognition.

Deep LSTM Architectures

The researchers investigated several novel ways to add spatial depth to an LSTM network, moving beyond simple recurrence to capture more complex hierarchical features:

  • Deep hidden-to-hidden transitions: Introducing projection layers such as LSTM-IP (Input Projection) and LSTM-OP (Output Projection) to refine internal state transitions.
  • Deep input-to-hidden functions: Feeding the output of standard feed-forward layers (e.g., ReLU layers) into the LSTM.
  • Deep hidden-to-output functions: Placing feed-forward layers on top of the LSTM layer before the final softmax output.
  • Stacked LSTMs: Directly stacking multiple LSTM layers to create a deep recurrent hierarchy.
Proposed deep LSTM architectural variants including stacked layers and feed-forward integrations.

Training and Parallelization

Training deep LSTMs is computationally intensive. To address this, the authors implemented a multi-GPU training setup using Truncated Back-Propagation Through Time (BPTT) and Asynchronous Stochastic Gradient Descent (ASGD). By splitting long utterances into overlapping subsequences, they were able to process multiple data streams in parallel across four GPUs. This approach proved that the traditionally slow LSTM training process can be efficiently distributed, making the training of deep recurrent models viable for large-scale tasks.

Experimental Results

The models were evaluated on the HKUST Mandarin Chinese Conversational Telephone Speech dataset using Character Error Rate (CER). The results clearly indicate that spatial depth is crucial for LSTM performance.

Comparison of Character Error Rates (CER) across various LSTM-based architectures.

Key findings from the experiments include:

  • Shallow LSTMs are insufficient: A standard 1-layer LSTM achieved a 40.28% CER, performing worse than standard feed-forward DNN baselines (e.g., ReLU DNN at 38.42%).
  • Benefits of Stacking: A 3-layer stacked LSTM significantly improved performance, dropping the error to 35.91%.
  • Efficiency of Projections: The LSTM-OP (Output Projection) architecture achieved a highly competitive 35.92% CER, matching the performance of the 3-layer stack with lower computational overhead.
  • Hybrid Superiority: The best results were obtained by combining recurrent and feed-forward depth. An LSTM-OP followed by three ReLU layers reached a state-of-the-art CER of 34.65%, representing an 8.87% relative error reduction over the best baseline DNNs.

Conclusion

This study demonstrates that deep LSTM architectures significantly outperform both shallow LSTMs and standard feed-forward DNNs in large vocabulary speech recognition. By systematically adding depth across different dimensions—input, hidden, and output—researchers can capture more complex acoustic patterns. Future work will explore even more complex combinations, such as integrating "maxout" activation units and further optimizing the interaction between recurrent and non-recurrent layers to push the boundaries of acoustic modeling.

Summary updated

Abstract

Long short-term memory (LSTM) based acoustic modeling methods have recently been shown to give state-of-the-art performance on some speech recognition tasks. To achieve a further performance improvement, in this research, deep extensions on LSTM are investigated considering that deep hierarchical model has turned out to be more efficient than a shallow one. Motivated by previous research on constructing deep recurrent neural networks (RNNs), alternative deep LSTM architectures are proposed and empirically evaluated on a large vocabulary conversational telephone speech recognition task. Meanwhile, regarding to multi-GPU devices, the training process for LSTM networks is introduced and discussed. Experimental results demonstrate that the deep LSTM networks benefit from the depth and yield the state-of-the-art performance on this task.
BibTeX
@inproceedings{li2015constructing,
  author = {Xiangang Li and Xihong Wu},
  title = {{Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=TLpC73IJes},
}
Figure