Constructing long short-term memory based deep recurrent neural networks for large vocabulary speech recognition

Xiangang Li, Xihong Wu

TL;DR

This paper explores various architectural strategies for building "deep" Long Short-Term Memory (LSTM) networks for large vocabulary speech recognition. By systematically adding depth to LSTMs across different dimensions—such as input-to-hidden, hidden-to-hidden, and hidden-to-output transitions—the authors demonstrate that deep LSTMs significantly outperform both shallow LSTMs and standard feed-forward Deep Neural Networks (DNNs), achieving state-of-the-art results on a Mandarin conversational telephone speech task.

What Problem Does This Paper Address?

At the time of this research, context-dependent Deep Neural Network Hidden Markov Models (CD-DNN-HMMs) were the dominant framework for acoustic modeling in speech recognition. However, because speech is inherently dynamic and sequential, Recurrent Neural Networks (RNNs)—specifically LSTMs—are theoretically better suited for the task.

While LSTMs are inherently "deep in time" (due to their recurrent nature), standard single-layer LSTMs often lack the "depth in space" (hierarchical representation) that makes feed-forward DNNs so powerful. The authors tackle the challenge of how to effectively extend LSTMs into deep architectures to capture more complex acoustic representations, while also managing the notoriously high computational cost of training them.

Methods

Drawing on theoretical work regarding deep RNNs, the authors propose and evaluate several novel ways to add depth to an LSTM network:

  • Deep hidden-to-hidden transition:
    • LSTM-IP (Input Projection): Adding a non-linear projection layer before the cell input activation.
  • LSTM-OP (Output Projection): Adding a separate projection layer after the LSTM memory blocks.
  • Deep input-to-hidden function: Feeding the output of standard feed-forward DNN layers (e.g., ReLU layers) into the LSTM.
  • Deep hidden-to-output function: Placing standard feed-forward DNN layers on top of the LSTM layer before the final softmax output.
  • Stacked LSTMs: The most straightforward approach, stacking multiple LSTM layers directly on top of one another.

To handle the massive computational requirements, the authors implemented a multi-GPU training setup using Truncated Back-Propagation Through Time (BPTT) and Asynchronous Stochastic Gradient Descent (ASGD), processing multiple subsequences in parallel across 4 GPUs.

Datasets

  • HKUST Mandarin Chinese Conversational Telephone Speech Recognition Corpus (LDC2005S15, LDC2005T32): A 150-hour dataset. The authors used ~10 hours for validation, the standard training set (873 calls) for network training, and the standard development set (24 calls) for testing.

Key Insights and Ideas

  • Shallow LSTMs aren't enough: A standard 1-layer LSTM actually performed worse than a standard 4-layer feed-forward DNN. Depth in space is strictly necessary to make LSTMs competitive in complex acoustic modeling.
  • Stacking isn't the only way to be deep: While stacking multiple LSTM layers works well, adding an Output Projection layer (LSTM-OP) achieves nearly identical performance to a 3-layer stacked LSTM, but with significantly less computational overhead.
  • Hybrid architectures win: The best performing models don't just rely on recurrent layers. Combining an LSTM-OP layer with several standard feed-forward layers (to deepen the hidden-to-output function) yielded the highest accuracy, proving that mixing recurrent and feed-forward depth is highly effective.
  • Parallelization is viable: By splitting utterances into overlapping subsequences and using ASGD, the authors proved that the traditionally slow LSTM training process can be efficiently distributed across multiple GPUs.

Key Results

The models were evaluated using Character Error Rate (CER).

  • Baselines: Standard GMM-HMM achieved 48.68% CER. Feed-forward DNN baselines hovered around 37-38% (e.g., ReLU DNN at 38.42%, Conv DNN at 37.13%).
  • Shallow LSTM: A standard 1-layer LSTM struggled, yielding a 40.28% CER.
  • Deep LSTMs:
    • Stacked LSTMs (3-layer) dropped the error to 35.91%.
  • LSTM-OP (Output Projection) achieved a highly competitive 35.92%.
  • The Best Model: Combining an LSTM-OP with 3 subsequent ReLU layers achieved the state-of-the-art result of 34.65% CER.
  • Overall Impact: The best deep LSTM architecture provided an 8.87% relative error reduction over the best baseline DNNs, and a massive 13.98% relative reduction compared to the shallow LSTM.

Limitations and Future Work

The authors frame this as a preliminary study and acknowledge a few limitations. Notably, some architectural combinations behaved unexpectedly—for example, placing an LSTM-OP layer on top of feed-forward intermediate layers yielded worse performance than using the LSTM-OP alone, a quirk that requires further investigation.

For future work, the authors suggest exploring even more complex combinations, such as LSTM-IP networks with multiple non-linear projection layers, stacked LSTMs interspersed with feed-forward layers, and integrating "maxout" activation units directly into the LSTM architecture to further boost performance.

Summary updated

Abstract

Long short-term memory (LSTM) based acoustic modeling methods have recently been shown to give state-of-the-art performance on some speech recognition tasks. To achieve a further performance improvement, in this research, deep extensions on LSTM are investigated considering that deep hierarchical model has turned out to be more efficient than a shallow one. Motivated by previous research on constructing deep recurrent neural networks (RNNs), alternative deep LSTM architectures are proposed and empirically evaluated on a large vocabulary conversational telephone speech recognition task. Meanwhile, regarding to multi-GPU devices, the training process for LSTM networks is introduced and discussed. Experimental results demonstrate that the deep LSTM networks benefit from the depth and yield the state-of-the-art performance on this task.
BibTeX
@inproceedings{li2015constructing,
  author = {Xiangang Li and Xihong Wu},
  title = {{Constructing long short-term memory based deep recurrent neural networks for large vocabulary speech recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=TLpC73IJes},
}

Loading related content...

Figure