Abstract
Acoustic models in real-time speech recognition systems typically stack multiple unidirectional LSTM layers to process acoustic frames over time. While prepending a frequency-LSTM (FLSTM) layer can improve performance by modeling time-frequency correlations, these architectures are often limited by a fixed, single-view window size and stride. We propose the Multi-view Frequency LSTM (mvFLSTM), which combines outputs from multiple FLSTM stacks with different views into a dimensionality-reduced feature representation. This architecture allows for modeling a wider range of time-frequency correlations. Evaluated on 50,000 hours of English far-field speech data, the mvFLSTM provides relative Word Error Rate (WER) improvements of 3–7% over optimized single-view FLSTM models while maintaining a similar computational footprint.
Introduction
Automatic Speech Recognition (ASR) in far-field scenarios—such as interacting with smart speakers in noisy environments—remains a significant challenge. Standard models utilize LSTMs to process speech temporally, but spectral distortions from background noise often degrade performance. Frequency LSTMs (FLSTMs) were introduced to address this by processing signals across the frequency axis before temporal layers. However, traditional FLSTMs use a fixed "view" (window size and stride), which may capture some spectral patterns while missing others. Simply increasing the size of these layers leads to a parameter explosion unsuitable for edge devices.
The Multi-view FLSTM Architecture
The core innovation of this work is the mvFLSTM frontend, which processes acoustic features through multiple parallel "lenses" along the frequency axis.
As shown in the architecture diagram above, the system consists of three primary components:
- Multi-view Processing: Multiple parallel FLSTM stacks utilize different window sizes ($F$) and strides ($S$). Narrow bands capture fine spectral details, while wider bands capture broader envelopes.
- Feature Concatenation: The outputs from these diverse views are concatenated into a single high-dimensional feature vector.
- Projection Layer (mvFLSTMp): To manage the increased data volume, a linear projection layer acts as a bottleneck. This layer squashes features back to a manageable size, serving as both a computational optimization and a noise filter that forces the model to learn compact, salient representations.
Experimental Setup
The models were trained on a massive internal Amazon dataset comprising 50,000 hours of English far-field speech using CTC loss followed by sMBR sequence training. Evaluation was performed on a test set of 37,000 utterances categorized to test specific robustness:
- Single-talker (ST) vs. Multi-talker (MT): Assessing performance in background chatter.
- Native (NT) vs. Non-native (nNT): Testing robustness across accents and speaking styles.
Key Results and Analysis
The multi-view approach demonstrated significant advantages over both standard LSTMs and highly optimized single-view FLSTMs.
As detailed in the results table, the best multi-view configuration (Model 13, mvFLSTMp) achieved a 12.61% relative WER reduction over the baseline LSTM and a 3–7% improvement over the best single-view FLSTM (Model 03). Notably, the gains were most pronounced in difficult categories like Multi-talker (MT) and Non-native (nNT) scenarios. Furthermore, the projection layer allowed for architectural decoupling; Model 11 actually reduced total parameters by 3.3% compared to the baseline while still delivering superior accuracy.
Conclusion
The mvFLSTM provides a simple yet effective method for enhancing ASR frontends. By leveraging diverse spectral perspectives and a bottleneck projection, the model achieves better robustness in noisy, far-field environments without increasing computational costs. Future work may explore automatically learning optimal window sizes and applying this frontend to modern end-to-end architectures like RNN-Transducers or Transformers.
Summary updated