Advancing Sequence-to-Sequence Based Speech Recognition

Zoltán Tüske, Kartik Audhkhasi, George SaonInterspeech 2019

TL;DR

This paper pushes the boundaries of end-to-end speech recognition by demonstrating that attention-based sequence-to-sequence models can rival and even surpass traditional hybrid systems. By systematically optimizing various components like data augmentation, dropout, and search strategies, the authors achieve state-of-the-art results on the challenging LibriSpeech benchmark, even without external language models.

What Problem Does This Paper Address?

Traditional Automatic Speech Recognition (ASR) systems often rely on complex hybrid architectures, combining acoustic models, pronunciation lexicons, and language models. While effective, these systems are intricate to train and maintain. End-to-end approaches, particularly those using sequence-to-sequence (seq2seq) models with attention mechanisms, offer a simpler pipeline by directly mapping audio to text. However, for a long time, these end-to-end models struggled to match the performance of the best hybrid systems, especially when augmented with powerful language models. This paper investigates whether seq2seq models, with careful optimization and advanced techniques, can close this performance gap and achieve state-of-the-art results on their own.

Methods

The core of the paper is the application and refinement of attention-based encoder-decoder models. The authors employ a standard architecture: a deep, pyramidal, bidirectional LSTM encoder to process audio features and an LSTM decoder that uses a location-aware attention mechanism to align the encoded audio with the output text sequence.

Key methodological advancements and optimizations include:

  • Data Augmentation: Extensive use of speed, tempo, and volume perturbation, along with injecting "sequence noise" (mixing in other utterances).
  • Regularization: Systematic tuning of dropout and drop-connect rates in the encoder and decoder.
  • Scheduled Sampling: A technique to gradually transition from using ground truth labels to model predictions during training, mitigating exposure bias.
  • Warm-Restart (SGDR): A learning rate annealing strategy that can help models escape local minima and achieve better performance.
  • Coverage Term: A mechanism in the attention mechanism to prevent it from attending to the same input frames repeatedly, improving alignment stability.
  • Prefix-Tree Search: Using a prefix-tree (trie) for decoding, which is crucial for efficient search with sub-word units.
  • Length Normalization: A technique to balance the probability of hypotheses of different lengths.
  • Vocabulary-Free Search: Exploring search strategies that don't strictly adhere to a predefined lexicon, potentially handling out-of-vocabulary words better.
  • Discriminative Training: Applying minimum word error rate (WER) training for a final fine-tuning step.

Datasets

The primary dataset used is LibriSpeech, a large, publicly available corpus of read English speech. It comprises approximately 1000 hours of training data, with development and test sets of about 10 hours each. LibriSpeech is further divided into "clean" and "other" subsets, with "other" being more challenging due to noise and speaking style variations. The authors also utilize a large text corpus (803 million running words) for training language models.

Key Insights and Ideas

  • Seq2Seq Models Can Compete: The most significant insight is that attention-based seq2seq models, with meticulous optimization, can achieve performance comparable to or even exceeding state-of-the-art hybrid systems, even without external language models.
  • Data Augmentation is Crucial: Perturbing the input audio (speed, tempo, volume) proved more impactful than injecting sequence noise, highlighting the importance of robust training data.
  • Coverage and Search Stability: The coverage term and prefix-tree search are vital for stabilizing the attention mechanism and ensuring reliable decoding, especially when scheduled sampling is reduced.
  • Smaller Units, Better Performance (with LM): For seq2seq models combined with language models, smaller modeling units (like BPE-100) often yield better results than larger units, suggesting that finer-grained modeling is beneficial.
  • Vocabulary-Free Search Potential: Allowing the search to go beyond a fixed vocabulary can offer slight gains and simplify the system, potentially by better handling unseen words.
  • Large LMs Still Matter: While seq2seq models are powerful, a large, well-trained language model still provides a significant boost in performance.

Key Results

  • State-of-the-Art WER: The paper reports achieving 4.0% WER on LibriSpeech test-clean and 11.7% on test-other without any external language model.
  • LM Integration: With a powerful recurrent language model, the WER drops to 2.7% on test-clean and 8.2% on test-other.
  • ROVER Combination: A simple ROVER combination of several attention-based systems achieved 2.5% WER on test-clean and 7.3% on test-other.
  • Outperforming Hybrid Systems: The best single system achieved results that challenge the best known hybrid models with language model rescoring.
  • Dropout Tuning: Optimal dropout and drop-connect values for the encoder were found to be around 0.2.
  • Beam Search Impact: For difficult "other" sets, optimal beam sizes can be significantly larger than commonly used, though this increases computational cost.

Limitations and Future Work

  • Computational Cost: While achieving excellent results, the paper notes that using very large beam sizes (over 100) with large language models can be an order of magnitude slower than real-time on a single CPU core.
  • Future Potential: The authors believe that with further advancements in training configurations and search parameters, even a single attention-based seq2seq system could reach even higher performance levels in the future. They suggest that exploring variations in model size, representation units, and input features can lead to further improvements.

Summary updated

BibTeX
@inproceedings{tuske2019advancing,
  author = {Zoltán Tüske and Kartik Audhkhasi and George Saon},
  title = {{Advancing Sequence-to-Sequence Based Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure