1-D Row-Convolution LSTM: Fast Streaming ASR at Accuracy Parity with LC-BLSTM

Kshitiz Kumar, Chaojun Liu, Yifan Gong, Jian Wu2020 · DOI

TL;DR

This paper introduces RC-LSTM, a novel acoustic model for Automatic Speech Recognition (ASR) that achieves state-of-the-art accuracy comparable to latency-controlled bidirectional LSTMs (LC-BLSTMs) but with significantly reduced latency and computational cost. By incorporating 1-dimensional row convolutions into standard LSTMs, RC-LSTM efficiently leverages future acoustic information, leading to faster and more efficient streaming ASR systems.

What Problem Does This Paper Address?

The field of Automatic Speech Recognition (ASR) is constantly striving for a delicate balance between accuracy, latency, and computational cost. While models like Bidirectional LSTMs (BLSTMs) offer high accuracy by considering both past and future speech information, they come with increased latency and computational demands, making them less ideal for real-time streaming applications. Latency-controlled BLSTMs (LC-BLSTMs) attempt to mitigate this by processing data in batches, but this can introduce discontinuities in hypotheses and still incurs significant computational overhead. The paper addresses the need for an ASR model that can achieve the high accuracy of BLSTMs while offering the low latency and computational efficiency required for responsive, real-time ASR systems.

Methods

The core innovation of this paper is the 1-Dimensional Row-Convolution LSTM (RC-LSTM). This model builds upon standard unidirectional LSTMs (UniLSTMs) by introducing a 1D row convolution operation. This operation allows the model to efficiently incorporate a limited amount of future acoustic information into each hidden layer.

Key aspects of the method include:

  • 1D Row Convolution: Instead of the typical approach where future information might be applied at the feature level or only to the top hidden layer, RC-LSTM uniformly distributes this future context across all hidden layers. This is achieved through a row convolution that connects the current hidden layer's output to the next layer's input, effectively creating a bottleneck that implicitly factorizes training into time and "frequency" dimensions.
  • Parameter Efficiency: This approach significantly reduces the number of parameters compared to methods that might splice future hidden states, leading to more compact models. The paper argues this reduces parameters from O(N²) to O(N).
  • Self-Attention Initialization: A novel self-attention initialization strategy for the convolution parameters is proposed, which is found to be critical for the convergence of models with deep lookaheads.
  • Uniform Lookahead: The authors advocate for a uniform lookahead parameter across all hidden layers, simplifying model design and eliminating the need for complex parameter tuning related to layer trajectories.

Datasets

The experiments were conducted on en-US tasks using two datasets:

  • Medium scale: 6,000 hours of training data.
  • Large scale: 60,000 hours of training data.

The evaluation on the medium scale task involved a near-field dataset of over 30 hours of speech across various scenarios (close-talk, far-talk, natural conversation, command-and-control, dictation, voice search, call center). The large-scale evaluation used over 200 hours of task data.

Key Insights and Ideas

  • Future Information is Key, Even in UniLSTMs: The paper reinforces the idea that incorporating future acoustic information is crucial for high-accuracy ASR, even when starting from a simpler unidirectional LSTM architecture.
  • Uniform Lookahead Simplifies Design: Distributing future context uniformly across all hidden layers, rather than selectively, simplifies the model architecture and training process without sacrificing performance. This contrasts with more complex approaches that might require careful tuning of lookahead parameters for different layers.
  • Implicit Factorization for Efficient Learning: The 1D row convolution implicitly factorizes the network into time and "frequency" dimensions. This orthogonal attribute is hypothesized to lead to more efficient learning, especially on large-scale tasks.
  • Compactness Enables Adaptation: The compact nature of the RC-LSTM's convolution parameters makes it well-suited for adaptation techniques, as demonstrated in experiments with reverberant environments.

Key Results

  • Accuracy Parity with LC-BLSTM: On a large-scale task (60k hours training), RC-LSTM achieved accuracy parity with LC-BLSTM, with WERs of 10.9% and 10.8% respectively.
  • Significant Latency Reduction: RC-LSTM demonstrated a 20% relative reduction in algorithmic latency compared to LC-BLSTM (480ms vs. 600ms). Crucially, the first ASR response latency was reduced by over 320ms (480ms vs. 800ms).
  • Computational Cost Reduction: RC-LSTM's computational cost was found to be only 50% of that for LC-BLSTM, making it much quicker to evaluate.
  • WER Reduction over UniLSTM: RC-LSTM achieved a 16% relative reduction in Word Error Rate (WER) compared to a standard UniLSTM model.
  • Improved Performance with Lookahead: Incorporating future information at the ASR feature level improved WER by 5.6% (relative to baseline B). Further improvements were seen by incorporating lookahead at hidden layers, with RC1 (T=1 for all layers) outperforming a system with all future context at the feature level. RC4 (T=4) showed a strong 16.2% WERR over the baseline.

Limitations and Future Work

  • Reverberant Conditions: While the paper shows adaptation capabilities in reverberant environments, the performance drop suggests that further work is needed to fully address such conditions. The authors note that the parameters become less confident in reverberant environments.
  • End-to-End ASR and Other Applications: The authors suggest that the simplicity and compactness of the RC operation could have significant applications for end-to-end ASR models and other domains beyond speech recognition.
  • Bidirectional RC-LSTM: The paper briefly mentions exploring a bidirectional version of RC-LSTM, which showed a 1% WERR improvement for RC4, indicating potential for further gains by combining past and future context within the RC framework.

Summary updated

BibTeX
@misc{kumar2020d,
  author = {Kshitiz Kumar and Chaojun Liu and Yifan Gong and Jian Wu},
  title = {{1-D Row-Convolution LSTM: Fast Streaming ASR at Accuracy Parity with LC-BLSTM}},
  year = {2020},
  doi = {10.21437/interspeech.2020-2894},
}
Figure