Abstract: In contrast to the huge success of self-attention based neural networks in various NLP tasks, the efficacy of self-attention in speech applications is yet limited. This is partly because the full effectiveness of the self-attention mechanism could not be achieved without proper down-sampling schemes in speech tasks. To address this issue, we propose a new self-attention mechanism suitable for speech recognition, namely, multi-stride self-attention. The proposed multi-stride approach lets each group of heads in self-attention process speech frames with a unique stride over neighboring frames. Thus, the entire attention mechanism would not be confined in a fixed frame shift and can have diverse contextual views for a given frame to determine attention weights more effectively. To validate our proposal we evaluated it on various speech corpora for speech recognition, both English and Chinese, and observed a consistent improvement, especially in terms of substitution and deletion errors, without the increase of model complexity. The average WER improvement of 7.5% (relative) obtained by the TDNNs having the multi-stride self-attention layer as compared to the baseline TDNN model shows the effectiveness of the proposed multi-stride self-attention mechanism.
Introduction
While self-attention has revolutionized Natural Language Processing (NLP), its success in Automatic Speech Recognition (ASR) has been more modest. The primary hurdle lies in the nature of speech data. Unlike words in a sentence, speech frames are typically sampled every 10ms, leading to high redundancy where neighboring frames are highly correlated. Furthermore, processing every single frame with standard self-attention is computationally expensive and often inefficient because the model can become bogged down in local similarities.
Previous solutions utilized fixed "down-sampling" or frame concatenation to reduce sequence length. However, using a single, fixed stride limits the model's perspective—it either sees too much detail (short stride) or misses fine-grained phonemic transitions (long stride).
Proposed Method: Multi-Stride Self-Attention
To overcome these limitations, we introduce Multi-Stride Self-Attention. Instead of having all heads in a multi-head attention layer look at the data through a single lens, we divide the heads into groups that process input speech frames using unique strides ($f_s$).
Key Architectural Features
- Parallel Pipelines: Each group of heads uses a different stride (e.g., $f_s=1, 3, 5$). Because each group uses a different stride but a fixed window size, they effectively cover different temporal scopes (e.g., 50ms, 150ms, and 250ms). This allows the model to capture both sharp phonemic changes and broader acoustic trends simultaneously.
- Transformer-style Block: We adopt the standard Transformer block structure—incorporating Layer Normalization, skip connections, and position-wise feed-forward networks. This structure performed significantly better than simpler "time-restricted" attention mechanisms.
- Parameter Efficiency: To ensure performance gains stem from the architecture rather than increased size, we reduced the hidden dimensions of the feed-forward layers, keeping the model size nearly constant.
Experimental Results
We evaluated the model on diverse benchmarks, including TED-LIUM (English broadcast), HUB5 (conversational telephone), Librispeech (audiobooks), and AISHELL-2 (Mandarin Chinese). Replacing the final layer of a standard Factorized Time-Delay Neural Network (TDNN-F) with a multi-stride self-attention layer yielded consistent improvements.
As shown in the table above, the multi-stride approach ($f_s = 1, 3, 5$) consistently outperforms any single-stride configuration. On the TED-LIUM test set, the Word Error Rate (WER) dropped from 8.7% (baseline) to 8.0%. Across all datasets, the proposed method achieved an average 7.5% relative WER improvement compared to the baseline TDNN-F model, with a negligible increase in parameters (from 10M to 11M).
Conclusion
The multi-stride approach is particularly effective at reducing substitution and deletion errors, suggesting the model is better at distinguishing similar-sounding phonemes and capturing essential acoustic cues. While this study focused on a hybrid DNN/HMM framework, the principles of multi-stride attention are expected to translate effectively to fully end-to-end encoder-decoder architectures.
Summary updated