Abstract
Traditional Automatic Speech Recognition (ASR) models often rely on stacked Long Short-Term Memory (LSTM) networks that simultaneously handle temporal modeling and phonetic classification. This paper introduces the Layer Trajectory BLSTM (ltBLSTM), which decouples these tasks by introducing a dedicated "depth-LSTM" to scan the outputs of a standard bidirectional LSTM across layers. Evaluated on large-scale Microsoft internal datasets, the ltBLSTM achieves up to a 14.5% relative reduction in Word Error Rate (WER), demonstrating superior performance and generalization capabilities.
Introduction
In the domain of Automatic Speech Recognition, bidirectional LSTMs (BLSTMs) have become a standard for capturing long-range temporal context. However, a fundamental limitation of standard stacked BLSTMs is that each cell must perform two distinct functions: temporal modeling (integrating past and future context) and classification (mapping features to specific phonetic units or senones). The authors hypothesize that this dual responsibility is suboptimal and propose a "Layer Trajectory" approach to separate these concerns.
The ltBLSTM Architecture
The proposed ltBLSTM architecture consists of two primary components: a Time-BLSTM and a Depth-LSTM. The Time-BLSTM is a multi-layer bidirectional network focused entirely on capturing temporal dependencies across the speech signal. Instead of passing the final layer's output directly to a softmax classifier, the ltBLSTM employs a Depth-LSTM at each time step $t$ to scan the hidden states across all $L$ layers of the Time-BLSTM.
This vertical scan allows the model to extract more refined features for classification, effectively specializing the network's layers. The Time-BLSTM determines "where" the signal is in the sequence, while the Depth-LSTM determines "what" phonetic unit is present. This specialization leads to richer feature representations compared to standard stacked architectures.
Design Variations
The researchers explored three variations of the depth component to optimize information flow:
- ltBLSTM (1LT): A single Depth-LSTM scans the concatenated forward and backward outputs of the Time-BLSTM.
- ltBLSTM (2LT): Two independent Depth-LSTMs scan the forward and backward paths separately, with their outputs combined only at the final output layer.
- ltBLSTM (2LT-concat): Two Depth-LSTMs where outputs are concatenated and shared at every hidden layer of the depth scan.
Interestingly, the simplest 1LT design proved highly robust, often matching or exceeding the performance of more complex multi-path designs. This suggests that the Depth-LSTM is highly efficient at distilling information from both temporal directions simultaneously.
Experimental Results
The models were trained on 30,000 hours of anonymized Microsoft internal data and evaluated across diverse scenarios, including short voice searches (Cortana), human-to-human conversations, and meeting data (DMA).
The ltBLSTM demonstrated significant improvements over standard BLSTMs:
- Word Error Rate Reduction: A relative WER reduction of 14.5% was observed on the challenging DMA meeting dataset, and 8.0% on conversational speech.
- Generalization: The substantial gains on the out-of-domain DMA dataset suggest that the decoupled architecture learns more fundamental acoustic features that generalize better to unseen environments.
- Comparison to State-of-the-Art: The ltBLSTM outperformed the previous uni-directional contextual layer trajectory LSTM (cltLSTM) by up to 12.7% relative WER, benefiting from its ability to see the entire future of an utterance.
Conclusion
The Layer Trajectory BLSTM represents a significant advancement in ASR by architectural specialization. By decoupling temporal modeling from classification, the model achieves state-of-the-art accuracy on large-scale real-world datasets. While the architecture introduces a 33% increase in parameters and associated computational overhead, the resulting gains in accuracy and generalization make it a compelling choice for high-performance speech recognition systems.
Summary updated