Transformer-Based End-to-End Speech Recognition with Residual Gaussian-Based Self-Attention

Chengdong Liang, Menglong Xu, Xiao-Lei Zhang

TL;DR

This paper tackles the "insufficient localness modeling" problem in Transformer-based speech recognition, where standard self-attention struggles with long sequences by overly focusing on global dependencies. They introduce novel attention mechanisms – Relative-Position-Aware Self-Attention (RPSA), Gaussian-Based Self-Attention (GSA), and Residual Gaussian-Based Self-Attention (resGSA) – that dynamically adapt to capture crucial local acoustic features, leading to improved accuracy.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems, particularly those based on the Transformer architecture, rely heavily on self-attention (SA) mechanisms. SA excels at modeling long-range dependencies by calculating pairwise similarities between input sequence elements. However, when applied to lengthy audio sequences, this global focus can be detrimental. The weighted averaging inherent in SA can cause the attention distribution to "disperse," effectively ignoring the subtle but critical relationships between adjacent audio frames. This "insufficient localness modeling" problem is exacerbated in long utterances, leading to reduced ASR accuracy. Existing solutions like fixed-window masking or relative positional embeddings have limitations, either being too rigid or not sufficiently adaptive to the dynamic nature of speech.

Methods

The authors propose three novel attention mechanisms to enhance local context modeling in Transformers for ASR:

  • Relative-Position-Aware Self-Attention (RPSA): This method modifies the standard SA by introducing a "neighboring edge connection" that explicitly strengthens the influence of nearby frames. It aims to balance global dependency modeling with improved local context awareness.
  • Gaussian-Based Self-Attention (GSA): To overcome the rigidity of RPSA's fixed window length, GSA introduces a learnable, adaptive Gaussian distribution. This allows the attention window to dynamically adjust its size and weights based on the specific input sequence, better capturing frame-dependent local features. The mean and variance of this Gaussian are learned from the input features themselves.
  • Residual Gaussian-Based Self-Attention (resGSA): This is an extension of GSA that further boosts performance by incorporating the attention scores from the previous layer as an additive residual. This residual connection helps the model converge more effectively and leverage information from prior attention computations.

These proposed attention mechanisms are integrated into the encoder of a Transformer-based ASR model.

Datasets

The primary dataset used for training and evaluation in this paper is the AISHELL-1 Mandarin speech corpus. This corpus contains over 170 hours of Mandarin speech data from 400 speakers, with standard splits for training (120,098 utterances), validation (14,326 utterances), and testing (7,176 utterances).

Key Insights and Ideas

  • Localness is King (for ASR): The paper strongly reinforces the idea that for speech recognition, capturing fine-grained local acoustic patterns is as, if not more, important than modeling very long-range dependencies. Standard SA's global bias is a bottleneck here.
  • Dynamic Adaptation is Crucial: The move from fixed windows (RPSA) to dynamically learned Gaussian windows (GSA) is a key insight. Speech signals are not uniform; different parts of an utterance might require different levels of local focus.
  • Residual Connections Pay Off: The resGSA demonstrates that building upon previous attention computations via residual connections can lead to more robust and accurate models, a common theme in deep learning architectures.
  • No Manual Tuning Needed: A significant advantage of GSA and resGSA is their ability to learn window parameters automatically, saving considerable effort in hyperparameter tuning compared to methods with fixed windows.

Key Results

The proposed methods show significant improvements on the AISHELL-1 Mandarin ASR corpus:

  • The resGSA-Transformer achieved the best performance, with a character error rate (CER) of 5.86% on the test set.
  • This represents a 7.8% relative reduction in CER compared to the baseline SA-Transformer.
  • The GSA-Transformer outperformed the RPSA-Transformer, indicating the benefit of dynamic window adaptation.
  • The resGSA-Transformer also outperformed other representative ASR systems, including TDNN-Chain, LAS, Self-attention Transducer, Speech-Transformer, and HA-Transformer, achieving a CER of 5.86% which is better than the HA-Transformer's 6.18%.
  • Crucially, the resGSA-Transformer achieved these gains without significantly increasing the number of parameters or computational complexity compared to the SA-Transformer.

Limitations and Future Work

  • Window Length Sensitivity: While GSA and resGSA are adaptive, the paper notes that RPSA's performance is sensitive to the choice of window length m, requiring manual tuning. Figure 5 illustrates this, showing CER decreasing and then stabilizing as m increases. The authors set m=30 for RPSA in their comparisons.
  • Future Work: The paper implicitly suggests that further exploration of adaptive attention mechanisms, potentially combining different types of local and global attention, could yield even better results. The success of residual connections also opens avenues for exploring other residual variants.

Summary updated

Abstract

Self-attention (SA), which encodes vector sequences according to their pairwise similarity, is widely used in speech recognition due to its strong context modeling ability. However, when applied to long sequence data, its accuracy is reduced. This is caused by the fact that its weighted average operator may lead to the dispersion of the attention distribution, which results in the relationship between adjacent signals ignored. To address this issue, in this paper, we introduce relative-position-awareness self-attention (RPSA). It not only maintains the global-range dependency modeling ability of self-attention, but also improves the localness modeling ability. Because the local window length of the original RPSA is fixed and sensitive to different test data, here we propose Gaussian-based self-attention (GSA) whose window length is learnable and adaptive to the test data automatically. We further generalize GSA to a new residual Gaussian self-attention (resGSA) for the performance improvement. We apply RPSA, GSA, and resGSA to Transformer-based speech recognition respectively. Experimental results on the AISHELL-1 Mandarin speech recognition corpus demonstrate the effectiveness of the proposed methods. For example, the resGSA-Transformer achieves a character error rate (CER) of 5.86% on the test set, which is relative 7.8% lower than that of the SA-Transformer. Although the performance of the proposed resGSA-Transformer is only slightly better than that of the RPSA-Transformer, it does not have to tune the window length manually.
BibTeX
@misc{liang2021transformer,
  author = {Chengdong Liang and Menglong Xu and Xiao-Lei Zhang},
  title = {{Transformer-Based End-to-End Speech Recognition with Residual Gaussian-Based Self-Attention}},
  year = {2021},
  eprint = {2103.15722},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.15722},
}
Figure