Layer Trajectory BLSTM

Eric Sun, Jinyu Li, Yifan Gong

Abstract

Traditional Automatic Speech Recognition (ASR) models often rely on stacked Long Short-Term Memory (LSTM) networks that simultaneously handle temporal modeling and phonetic classification. This paper introduces the Layer Trajectory BLSTM (ltBLSTM), which decouples these tasks by introducing a dedicated "depth-LSTM" to scan the outputs of a standard bidirectional LSTM across layers. Evaluated on large-scale Microsoft internal datasets, the ltBLSTM achieves up to a 14.5% relative reduction in Word Error Rate (WER), demonstrating superior performance and generalization capabilities.

Introduction

In the domain of Automatic Speech Recognition, bidirectional LSTMs (BLSTMs) have become a standard for capturing long-range temporal context. However, a fundamental limitation of standard stacked BLSTMs is that each cell must perform two distinct functions: temporal modeling (integrating past and future context) and classification (mapping features to specific phonetic units or senones). The authors hypothesize that this dual responsibility is suboptimal and propose a "Layer Trajectory" approach to separate these concerns.

The ltBLSTM Architecture

The proposed ltBLSTM architecture consists of two primary components: a Time-BLSTM and a Depth-LSTM. The Time-BLSTM is a multi-layer bidirectional network focused entirely on capturing temporal dependencies across the speech signal. Instead of passing the final layer's output directly to a softmax classifier, the ltBLSTM employs a Depth-LSTM at each time step $t$ to scan the hidden states across all $L$ layers of the Time-BLSTM.

The ltBLSTM architecture decouples temporal modeling (Time-BLSTM) from classification features (Depth-LSTM).

This vertical scan allows the model to extract more refined features for classification, effectively specializing the network's layers. The Time-BLSTM determines "where" the signal is in the sequence, while the Depth-LSTM determines "what" phonetic unit is present. This specialization leads to richer feature representations compared to standard stacked architectures.

Design Variations

The researchers explored three variations of the depth component to optimize information flow:

  1. ltBLSTM (1LT): A single Depth-LSTM scans the concatenated forward and backward outputs of the Time-BLSTM.
  2. ltBLSTM (2LT): Two independent Depth-LSTMs scan the forward and backward paths separately, with their outputs combined only at the final output layer.
  3. ltBLSTM (2LT-concat): Two Depth-LSTMs where outputs are concatenated and shared at every hidden layer of the depth scan.
Comparison of dual-path depth-LSTM architectures: (a) 2LT and (b) 2LT-concat.

Interestingly, the simplest 1LT design proved highly robust, often matching or exceeding the performance of more complex multi-path designs. This suggests that the Depth-LSTM is highly efficient at distilling information from both temporal directions simultaneously.

Experimental Results

The models were trained on 30,000 hours of anonymized Microsoft internal data and evaluated across diverse scenarios, including short voice searches (Cortana), human-to-human conversations, and meeting data (DMA).

The ltBLSTM demonstrated significant improvements over standard BLSTMs:

  • Word Error Rate Reduction: A relative WER reduction of 14.5% was observed on the challenging DMA meeting dataset, and 8.0% on conversational speech.
  • Generalization: The substantial gains on the out-of-domain DMA dataset suggest that the decoupled architecture learns more fundamental acoustic features that generalize better to unseen environments.
  • Comparison to State-of-the-Art: The ltBLSTM outperformed the previous uni-directional contextual layer trajectory LSTM (cltLSTM) by up to 12.7% relative WER, benefiting from its ability to see the entire future of an utterance.

Conclusion

The Layer Trajectory BLSTM represents a significant advancement in ASR by architectural specialization. By decoupling temporal modeling from classification, the model achieves state-of-the-art accuracy on large-scale real-world datasets. While the architecture introduces a 33% increase in parameters and associated computational overhead, the resulting gains in accuracy and generalization make it a compelling choice for high-performance speech recognition systems.

Summary updated

Abstract

Recently, we proposed layer trajectory (LT) LSTM (ltLSTM) which significantly outperforms LSTM by decoupling the functions of senone classification and temporal modeling with separate depth and time LSTMs. We further improved ltLSTM with contextual layer trajectory LSTM (cltLSTM) which uses the future context frames to predict target labels. Given bidirectional LSTM (BLSTM) also uses future context frames to improve its modeling power, in this study we first compare the performance between these two models. Then we apply the layer trajectory idea to further improve BLSTM models, in which BLSTM is in charge of modeling the temporal information while depth-LSTM takes care of senone classification. In addition, we also investigate the model performance among different LT component designs on BLSTM models. Trained with 30 thousand hours of EN-US Microsoft internal data, the proposed layer trajectory BLSTM (ltBLSTM) model improved the baseline BLSTM with up to 14.5% relative word error rate (WER) reduction across different tasks.
BibTeX
@inproceedings{sun2019layer,
  author = {Eric Sun and Jinyu Li and Yifan Gong},
  title = {{Layer Trajectory BLSTM}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2971},
}
Figure