TL;DR
This paper introduces novel unidirectional neural network architectures, TDLSTM and PTDLSTM, designed for end-to-end automatic speech recognition (ASR) systems that require low latency for streaming applications. These new architectures achieve state-of-the-art results on several ASR tasks, outperforming existing unidirectional models and narrowing the performance gap with non-streaming bidirectional models.
What Problem Does This Paper Address?
Traditional hybrid ASR systems rely on separate acoustic models, language models, and pronunciation dictionaries. End-to-end ASR aims to simplify this by merging all components into a single neural network. A key challenge in end-to-end ASR, especially for streaming applications (like real-time voice assistants), is processing temporal information effectively while maintaining low latency. Bidirectional LSTMs (BLSTMs) have shown excellent performance but are unsuitable for streaming due to their reliance on future context. While latency-controlled BLSTMs (LCBLSTMs) address this, they incur computational overhead. Existing unidirectional LSTMs are suitable for streaming but may not capture temporal dynamics as effectively as bidirectional models. This paper tackles the need for unidirectional architectures that can achieve high ASR performance without sacrificing low latency.
Methods
The authors propose two new unidirectional neural network building blocks for encoder architectures:
- Time-Delay LSTM (TDLSTM): This architecture reverses the typical order of feed-forward and LSTM layers. It first processes time-delayed and concatenated inputs with an LSTM, followed by a feed-forward bottleneck layer. This bottleneck layer reduces the dimensionality, aiming to decrease computational cost and improve efficiency.
- Parallel Time-Delayed LSTM (PTDLSTM): Inspired by the parallel nature of bidirectional LSTMs, this approach processes each time-delayed input stream with a separate, non-shared LSTM layer. The outputs of these parallel LSTMs are then concatenated and processed through a bottleneck layer.
These building blocks are integrated into a deep time-delay neural network architecture. The overall architecture is designed to capture a significant temporal context (25 frames before and 25 frames after the center frame) while ensuring a fixed, low processing latency. The models are trained using a hybrid CTC/attention objective function.
Datasets
The experiments were conducted on three publicly available datasets:
- Wall Street Journal (WSJ): An English corpus of read newspapers, approximately 80 hours of training data.
- LibriSpeech: An English corpus derived from public domain audiobooks, featuring various recording qualities, with approximately 960 hours of training data.
- HKUST: A Mandarin telephone speech corpus, with approximately 174 hours of training data.
Key Insights and Ideas
- Unidirectional models can achieve near-bidirectional performance: The paper demonstrates that carefully designed unidirectional architectures can significantly close the performance gap with powerful but non-streaming bidirectional models.
- Bottleneck layers are effective for efficiency: The inclusion of bottleneck layers in TDLSTM and PTDLSTM helps reduce model complexity and potentially computational cost without a significant drop in performance.
- Parallel processing of time-delayed streams is beneficial: The PTDLSTM architecture's success suggests that parallelizing the processing of different temporal contexts within a unidirectional framework is a promising direction.
- Order matters in TDLSTM: The authors highlight that reversing the order of LSTM and feed-forward layers in TDLSTM leads to improved results compared to a standard TDNN-LSTM.
Key Results
The proposed TDLSTM and PTDLSTM architectures demonstrated improved ASR performance across all tested tasks (WSJ, HKUST, and LibriSpeech) compared to baseline unidirectional LSTM and TDNN-LSTM models.
- The TDLSTM architecture improved error rates by an average of 0.3% compared to the best baseline (LCBLSTM).
- The PTDLSTM architecture achieved the lowest error rates among all tested streaming architectures.
- On the LibriSpeech dataset, the PTDLSTM model achieved an average relative word error rate improvement of 12.3% compared to the baseline LSTM.
- The PTDLSTM architecture significantly reduced the performance gap towards the non-streaming BLSTM model.
- The PTDLSTM model offered advantages in training and inference speed due to its entirely unidirectional nature.
Limitations and Future Work
- LibriSpeech WER deterioration for TDNN-LSTM: The paper notes that the word error rates for the "Kaldi"-TDNN-LSTM model on the LibriSpeech task were deteriorated, and the reason is not immediately obvious, requiring further investigation.
- Future work: The authors mention that combining their proposed architectures with triggered attention mechanisms for streaming recognition is a direction for future work. They also suggest that deeper investigation into the LibriSpeech results for TDNN-LSTM is warranted.
Summary updated