Multi-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition

Maarten Van Segbroeck, Sri Harish Mallidi, Brian King, I-Fan Chen, Gurpreet Chadha, Roland MaasarXiv 2020 · OpenReview

Abstract

Acoustic models in real-time speech recognition systems typically stack multiple unidirectional LSTM layers to process acoustic frames over time. While prepending a frequency-LSTM (FLSTM) layer can improve performance by modeling time-frequency correlations, these architectures are often limited by a fixed, single-view window size and stride. We propose the Multi-view Frequency LSTM (mvFLSTM), which combines outputs from multiple FLSTM stacks with different views into a dimensionality-reduced feature representation. This architecture allows for modeling a wider range of time-frequency correlations. Evaluated on 50,000 hours of English far-field speech data, the mvFLSTM provides relative Word Error Rate (WER) improvements of 3–7% over optimized single-view FLSTM models while maintaining a similar computational footprint.

Introduction

Automatic Speech Recognition (ASR) in far-field scenarios—such as interacting with smart speakers in noisy environments—remains a significant challenge. Standard models utilize LSTMs to process speech temporally, but spectral distortions from background noise often degrade performance. Frequency LSTMs (FLSTMs) were introduced to address this by processing signals across the frequency axis before temporal layers. However, traditional FLSTMs use a fixed "view" (window size and stride), which may capture some spectral patterns while missing others. Simply increasing the size of these layers leads to a parameter explosion unsuitable for edge devices.

The Multi-view FLSTM Architecture

The core innovation of this work is the mvFLSTM frontend, which processes acoustic features through multiple parallel "lenses" along the frequency axis.

The mvFLSTM architecture featuring parallel FLSTM stacks followed by a projection layer.

As shown in the architecture diagram above, the system consists of three primary components:

  1. Multi-view Processing: Multiple parallel FLSTM stacks utilize different window sizes ($F$) and strides ($S$). Narrow bands capture fine spectral details, while wider bands capture broader envelopes.
  2. Feature Concatenation: The outputs from these diverse views are concatenated into a single high-dimensional feature vector.
  3. Projection Layer (mvFLSTMp): To manage the increased data volume, a linear projection layer acts as a bottleneck. This layer squashes features back to a manageable size, serving as both a computational optimization and a noise filter that forces the model to learn compact, salient representations.

Experimental Setup

The models were trained on a massive internal Amazon dataset comprising 50,000 hours of English far-field speech using CTC loss followed by sMBR sequence training. Evaluation was performed on a test set of 37,000 utterances categorized to test specific robustness:

  • Single-talker (ST) vs. Multi-talker (MT): Assessing performance in background chatter.
  • Native (NT) vs. Non-native (nNT): Testing robustness across accents and speaking styles.

Key Results and Analysis

The multi-view approach demonstrated significant advantages over both standard LSTMs and highly optimized single-view FLSTMs.

Relative WER reduction (%) for various model configurations across different acoustic scenarios.

As detailed in the results table, the best multi-view configuration (Model 13, mvFLSTMp) achieved a 12.61% relative WER reduction over the baseline LSTM and a 3–7% improvement over the best single-view FLSTM (Model 03). Notably, the gains were most pronounced in difficult categories like Multi-talker (MT) and Non-native (nNT) scenarios. Furthermore, the projection layer allowed for architectural decoupling; Model 11 actually reduced total parameters by 3.3% compared to the baseline while still delivering superior accuracy.

Conclusion

The mvFLSTM provides a simple yet effective method for enhancing ASR frontends. By leveraging diverse spectral perspectives and a bottleneck projection, the model achieves better robustness in noisy, far-field environments without increasing computational costs. Future work may explore automatically learning optimal window sizes and applying this frontend to modern end-to-end architectures like RNN-Transducers or Transformers.

Summary updated

Abstract

Acoustic models in real-time speech recognition systems typically stack multiple unidirectional LSTM layers to process the acoustic frames over time. Performance improvements over vanilla LSTM architectures have been reported by prepending a stack of frequency-LSTM (FLSTM) layers to the time LSTM. These FLSTM layers can learn a more robust input feature to the time LSTM layers by modeling time-frequency correlations in the acoustic input signals. A drawback of FLSTM based architectures however is that they operate at a predefined, and tuned, window size and stride, referred to as 'view' in this paper. We present a simple and efficient modification by combining the outputs of multiple FLSTM stacks with different views, into a dimensionality reduced feature representation. The proposed multi-view FLSTM architecture allows to model a wider range of time-frequency correlations compared to an FLSTM model with single view. When trained on 50K hours of English far-field speech data with CTC loss followed by sMBR sequence training, we show that the multi-view FLSTM acoustic model provides relative Word Error Rate (WER) improvements of 3-7% for different speaker and acoustic environment scenarios over an optimized single FLSTM model, while retaining a similar computational footprint.
BibTeX
@misc{segbroeck2020multi,
  author = {Maarten Van Segbroeck and Sri Harish Mallidi and Brian King and I-Fan Chen and Gurpreet Chadha and Roland Maas},
  title = {{Multi-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition}},
  year = {2020},
  eprint = {2007.00131},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.00131},
}
Figure