Hybridformer: Improving Squeezeformer with Hybrid Attention and NSR Mechanism

Yuguang Yang, Yu Pan, Jingjing Yin, Jiangyu Han, Lei Ma, Heng Lu

Abstract

Automatic Speech Recognition (ASR) models like SqueezeFormer have significantly advanced the state-of-the-art by reducing temporal redundancy. However, they still face challenges related to the quadratic computational complexity of attention mechanisms and rigid local feature extraction. This paper introduces HybridFormer, an architecture that optimizes SqueezeFormer through two key innovations: the Linear and Softmax Attention (LASA) paradigm and the NAS-guided Structural Re-parameterization (NSR) mechanism. HybridFormer achieves a 9.1% relative reduction in Word Error Rate (WER) on the LibriSpeech test-other set and provides up to an 18% inference speedup for long audio sequences.

Introduction

Modern ASR architectures often rely on Softmax Attention (SA), which possesses a computational complexity of $O(N^2)$, where $N$ is the input sequence length. This quadratic bottleneck makes processing long audio clips computationally expensive. Furthermore, standard models like SqueezeFormer utilize fixed-size convolution kernels (e.g., 31x1) for local context modeling, which may not be optimal for capturing diverse acoustic patterns. HybridFormer addresses these limitations by dynamically balancing attention complexity and enhancing local modeling flexibility without increasing inference latency.

Methodology

1. The LASA Paradigm

To mitigate the quadratic bottleneck, HybridFormer employs a hybrid attention strategy called LASA (Linear and Softmax Attention). The choice of attention mechanism is tied to the sampling rate of the specific block:

  • Linear Attention (LA): Implemented in "unsqueeze" blocks (40ms sampling rate) where the sequence length is greatest. LA reduces complexity to $O(N)$, significantly accelerating the processing of long inputs.
  • Softmax Attention (SA): Reserved for "squeeze" blocks (80ms sampling rate) where the sequence is shorter, ensuring high-precision global modeling where it is most critical.
  • Rotary Position Embeddings (RoPE): To maintain spatial awareness across both attention types, the authors replaced standard relative position embeddings with RoPE, which integrates more effectively with Linear Attention.

2. NSR Mechanism

The NAS-guided Structural Re-parameterization (NSR) mechanism improves local feature extraction. During the training phase, Neural Architecture Search (NAS) is used to identify an optimal combination of smaller kernels (e.g., 1x1, 3x1, 5x1) that supplement the primary 31x1 convolution branch.

The HybridFormer architecture overview, detailing the transition from SqueezeFormer blocks to HybridFormer blocks incorporating NSR and LASA.

Crucially, these multiple branches are mathematically collapsed into a single 31x1 convolution during inference. This "structural re-parameterization" allows the model to benefit from multi-scale feature fusion during training while maintaining the inference speed of a simple single-branch model.

Experimental Results

Accuracy and Efficiency

Evaluated on the LibriSpeech benchmark, HybridFormer demonstrated superior performance compared to the baseline SqueezeFormer. It achieved a 9.1% relative WER reduction on the test-other subset. Notably, this accuracy gain was achieved with a slight reduction in total parameters (21.45M for HybridFormer vs. 22.22M for SqueezeFormer).

Inference Speed

The efficiency of the LASA paradigm is most evident when processing long audio sequences. For 30-second inputs, HybridFormer is 18% faster than SqueezeFormer. As shown in the inference time comparison, the speed advantage of HybridFormer over standard Conformer and SqueezeFormer models grows linearly with input length, effectively overcoming the quadratic scaling issues of pure Softmax Attention models.

Comparison of average inference time cost across varying input speech lengths for different ASR architectures.

Conclusion

HybridFormer represents a significant optimization of the SqueezeFormer architecture for ASR. By matching attention complexity to data resolution via the LASA paradigm and utilizing NSR for flexible local modeling, the model achieves a rare combination of improved accuracy and reduced computational latency. Future work will focus on reducing the complexity of the NAS-guided training stages and evaluating the architecture's robustness on more diverse, multi-lingual datasets.

Summary updated

BibTeX
@inproceedings{yang2023hybridformer,
  author = {Yuguang Yang and Yu Pan and Jingjing Yin and Jiangyu Han and Lei Ma and Heng Lu},
  title = {{Hybridformer: Improving Squeezeformer with Hybrid Attention and NSR Mechanism}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096467},
}
Figure