Speech Recognition with Deep Recurrent Neural Networks

Alex Graves, Abdel-rahman Mohamed, Geoffrey E. Hinton

TL;DR

This paper introduces "deep recurrent neural networks" (DRNNs) by stacking multiple Long Short-Term Memory (LSTM) layers, demonstrating a significant leap in speech recognition performance. By combining the temporal modeling power of RNNs with the hierarchical representations of deep learning, they achieve a new state-of-the-art on the TIMIT benchmark.

What Problem Does This Paper Address?

Traditional speech recognition systems often rely on combining Recurrent Neural Networks (RNNs) with Hidden Markov Models (HMMs). While deep feedforward networks have recently shown impressive results in acoustic modeling, RNNs, despite their natural ability to handle sequential data, have lagged behind in speech recognition. This paper investigates why RNNs haven't performed as well and proposes a novel architecture, deep bidirectional LSTMs, to overcome these limitations and leverage the strengths of both deep learning and recurrent architectures for speech recognition.

Methods

The core innovation is the concept of "deep recurrent neural networks," achieved by stacking multiple RNN layers. Specifically, they utilize the Long Short-Term Memory (LSTM) architecture, known for its ability to capture long-range dependencies, and extend it to be deep (multiple layers) and bidirectional (processing data in both forward and backward directions).

Key techniques include:

  • Deep LSTM: Stacking multiple LSTM layers, where the output of one layer becomes the input to the next, creating hierarchical temporal representations.
  • Bidirectional RNNs (BRNNs): Processing the input sequence in both forward and backward directions to capture context from both past and future. Combining this with LSTM results in deep bidirectional LSTMs.
  • End-to-End Training: Training the entire network directly from acoustic input to phonetic output, bypassing the need for pre-defined alignments and avoiding potential errors from incorrect alignments.
  • Connectionist Temporal Classification (CTC): A method for training RNNs on sequence labeling tasks where input-output alignment is unknown. It uses a softmax layer to predict phonemes (or a blank symbol) at each time step.
  • RNN Transducer: An extension that jointly trains acoustic and language models by considering both input sequence position and previously emitted outputs.
  • Regularization: Employing techniques like early stopping and weight noise to prevent overfitting, which is crucial for complex deep models.

Datasets

  • TIMIT Corpus: This is the primary dataset used for training and evaluation. The authors use a standard split: the 462-speaker set for training, a 50-speaker set for early stopping, and the 24-speaker core test set for reporting results.

Key Insights and Ideas

  • Depth Matters for RNNs in Speech: Just as depth has been crucial for feedforward networks, stacking RNN layers (creating deep RNNs) significantly improves speech recognition performance.
  • LSTM is Superior to Tanh for Speech: For this task, LSTM units proved much more effective than standard tanh units within the recurrent layers.
  • Bidirectionality is Key: Processing speech data in both forward and backward directions is essential for capturing the full context needed for accurate recognition.
  • End-to-End Training is Powerful: Training directly from acoustic features to phonetic labels simplifies the pipeline and avoids issues with alignment.
  • Pretraining Enhances Transducers: While RNN transducers can be trained from random weights, pretraining them with weights from a trained CTC network and a separate prediction network yields better results.

Key Results

The paper reports a new state-of-the-art phoneme error rate (PER) on the TIMIT core test set.

  • Deep CTC RNNs: Increasing the number of hidden layers from one to five in CTC-based networks reduced the error rate from 23.9% to 18.4%.
  • Best Performance: The deep bidirectional LSTM trained with CTC achieved a test set error rate of 17.7%, which was the best recorded score at the time of publication.
  • Transducer Advantage: While randomly initialized transducers performed similarly to CTC, pre-trained transducers achieved a PER of 17.7%, slightly outperforming CTC.

Limitations and Future Work

  • Single-Task Focus: The current work focuses on phoneme recognition. Extending the system to large vocabulary speech recognition is a natural next step.
  • Weight Noise Sensitivity: One network (CTC-31-500h-tanh) failed to learn when weight noise was applied, suggesting potential sensitivity to regularization techniques for certain configurations.
  • Future Directions: The authors suggest exploring the combination of deep LSTM with frequency-domain convolutional neural networks for speech recognition.

Summary updated

BibTeX
@inproceedings{graves2013speech,
  author = {Alex Graves and Abdel-rahman Mohamed and Geoffrey E. Hinton},
  title = {{Speech Recognition with Deep Recurrent Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2013},
  doi = {10.1109/icassp.2013.6638947},
}
Figure