Self-Attention Transducers for End-to-End Speech Recognition

Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Ye Bai, Zhengqi WenInterspeech 2019 · DOI · OpenReview

Abstract

Recurrent neural network transducers (RNN-T) have been successfully applied in end-to-end speech recognition. However, the recurrent structure makes it difficult for parallelization. In this paper, we propose a self-attention transducer (SA-T) for speech recognition. RNNs are replaced with self-attention blocks, which are powerful to model long-term dependencies inside sequences and able to be efficiently parallelized. Furthermore, a path-aware regularization is proposed to assist SA-T to learn alignments and improve the performance. Additionally, a chunk-flow mechanism is utilized to achieve online decoding. All experiments are conducted on a Mandarin Chinese dataset AISHELL-1. The results demonstrate that our proposed approach achieves a 21.3% relative reduction in character error rate compared with the baseline RNN-T. In addition, the SA-T with chunk-flow mechanism can perform online decoding with only a little degradation of the performance.

Introduction

The gold standard for end-to-end speech recognition has long been the Recurrent Neural Network Transducer (RNN-T). While effective, RNNs are inherently sequential—they process data one step at a time, which makes them difficult to parallelize and slow to train, especially on long audio clips. Furthermore, standard self-attention mechanisms (like the Transformer) usually require looking at an entire sentence at once to work effectively. This "global lookahead" makes it nearly impossible to use them for online decoding (streaming speech-to-text), where the system needs to produce words as the user is still speaking.

Methodology

The authors propose three key innovations to transform the transducer architecture into a parallelizable, high-performance system.

Self-Attention Transducer (SA-T)

The researchers replaced the RNNs in both the acoustic encoder and the linguistic prediction network with self-attention blocks. These blocks use Multi-Head Attention (MHA) to model long-distance relationships in the audio signal much more efficiently than RNNs. By removing the recurrent dependency, the SA-T can process all time steps simultaneously during training.

The SA-T architecture, internal self-attention block, and the output probability graph.

Path-Aware Regularization (PAR)

Transducers often struggle to learn the correct "alignment" (matching specific audio frames to specific characters) early in training. PAR uses a "gold standard" alignment path—generated via forced alignment from tools like Kaldi—to provide an extra supervisory signal. It essentially guides the model on which character to focus on at specific time intervals, which stabilizes training and forces the model to produce "sharper" probability distributions.

Chunk-Flow Mechanism for Online Decoding

To enable streaming, the authors introduced a sliding-window approach called the Chunk-Flow mechanism. Instead of attending to the whole utterance, each layer only looks at a small "chunk" of frames.

The chunk-flow mechanism for online decoding.

As illustrated above, each block attends to a fixed number of frames to the left ($N_l$) and right ($N_r$). By stacking these layers, the model gains a wide enough receptive field to understand context without needing the full future audio, making it viable for real-time use.

Experimental Results

The model was evaluated on AISHELL-1, a popular open-source Mandarin Chinese speech corpus containing approximately 150 hours of training data.

  • Accuracy: The proposed SA-T with PAR achieved a 9.30% Character Error Rate (CER) on the test set.
  • Comparison: This represents a 21.3% relative improvement over the baseline RNN-T (11.82% CER) and also outperformed the Listen, Attend and Spell (LAS) model (10.56% CER).
  • Streaming Performance: The Chunk-Flow mechanism proved that a window of 20 frames to the left and 10 to the right was sufficient to outperform the baseline RNN-T while remaining viable for real-time applications.

Conclusion

The Self-Attention Transducer offers a powerful, parallelizable alternative to RNN-based models. Path-Aware Regularization proved to be a critical "secret sauce," dropping the CER by an additional 11.1% relatively by providing alignment guidance. While the current implementation relies on external toolkits for initial alignments, future work aims to refine the chunk-flow mechanism and integrate more sophisticated neural language models to further reduce error rates and latency.

Summary updated

Abstract

Recurrent neural network transducers (RNN-T) have been successfully applied in end-to-end speech recognition. However, the recurrent structure makes it difficult for parallelization . In this paper, we propose a self-attention transducer (SA-T) for speech recognition. RNNs are replaced with self-attention blocks, which are powerful to model long-term dependencies inside sequences and able to be efficiently parallelized. Furthermore, a path-aware regularization is proposed to assist SA-T to learn alignments and improve the performance. Additionally, a chunk-flow mechanism is utilized to achieve online decoding. All experiments are conducted on a Mandarin Chinese dataset AISHELL-1. The results demonstrate that our proposed approach achieves a 21.3% relative reduction in character error rate compared with the baseline RNN-T. In addition, the SA-T with chunk-flow mechanism can perform online decoding with only a little degradation of the performance.
BibTeX
@inproceedings{tian2019self,
  author = {Zhengkun Tian and Jiangyan Yi and Jianhua Tao and Ye Bai and Zhengqi Wen},
  title = {{Self-Attention Transducers for End-to-End Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2203},
}
Figure