Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications

Yongqiang Wang, Yangyang Shi, Frank Zhang, Chunyang Wu, Julian Chan, Ching-Feng Yeh, Alex Xiao

Abstract

In this paper, we summarize the application of transformer and its streamable variant, Emformer based acoustic model for large scale speech recognition applications. We compare the transformer based acoustic models with their LSTM counterparts on industrial scale tasks. Specifically, we compare Emformer with latency-controlled BLSTM (LCBLSTM) on medium latency tasks and LSTM on low latency tasks. On a low latency voice assistant task, Emformer gets 24% to 26% relative word error rate reductions (WERRs). For medium latency scenarios, comparing with LCBLSTM with similar model size and latency, Emformer gets significant WERR across four languages in video captioning datasets with 2-3 times inference real-time factors reduction.

Introduction

The field of Automatic Speech Recognition (ASR) faces a persistent challenge: balancing high accuracy with low latency. While standard Transformer models have set new benchmarks for offline transcription due to their non-causal nature, they are often unsuitable for real-time applications like live captioning or voice assistants. Historically, Long Short-Term Memory (LSTM) networks were the industry standard for streaming tasks. However, LSTMs are difficult to parallelize and lack the modeling capacity of Transformers. This paper introduces Emformer (Efficient Memory Transformer), a streamable variant designed to bridge this gap by providing Transformer-level accuracy with the efficiency required for industrial-scale, real-time deployment.

Methods

The core of the Emformer architecture is a block-processing approach that slices audio into segments. To maintain contextual awareness, each segment incorporates a "right context" (look-ahead) and a "left context" (history). A key innovation in Emformer is its efficient caching mechanism. Unlike previous augmented memory transformers that recomputed historical context at every step, Emformer utilizes pre-computed keys and values from previous segments.

Emformer architecture illustrating the use of memory (M), left context (L), center (C), and right context (R) segments with efficient caching.

As shown in Figure 1, the model uses a memory bank (M) to summarize long-range history. This allows the model to capture distant dependencies without the quadratic computational overhead of standard self-attention. By caching the keys and values of the left context, the model avoids the redundant computation that plagued previous streaming transformer implementations. This architecture supports both traditional hybrid HMM-based systems and modern neural transducer architectures.

Experimental Setup

The researchers evaluated Emformer on massive, industrial-scale datasets, moving far beyond standard academic benchmarks:

  • Voice Assistant Task: A 47k-hour dataset (initially 12.5k hours of supervised data augmented with simulated noise and room acoustics).
  • Video ASR Task: Spanning four languages (English, Spanish, Vietnamese, and German) with supervised data ranging from 9k to 39k hours.
  • Large-Scale Semi-Supervised Learning: A "Giant" set comprising 2.2 million hours of English data to test the model's scaling limits.

Key Results and Insights

Emformer demonstrated substantial improvements over LSTM baselines across various latency requirements. In low-latency voice assistant tasks (~80-120ms), Emformer achieved a relative Word Error Rate Reduction (WERR) of 24% to 26% compared to standard LSTMs.

For medium-latency scenarios such as video captioning, Emformer was compared against Latency-Controlled Bidirectional LSTMs (LCBLSTM). As detailed in Figure 5, Emformer not only improved accuracy but also significantly reduced the Real-Time Factor (RTF), running 2-3 times faster than LCBLSTM across multiple languages.

Performance comparison of Emformer against LCBLSTM across multiple languages and systems, demonstrating significant RTF reductions.

The results also highlighted the impact of scale. Training on the 2.2 million-hour dataset provided a significant boost, particularly on "extreme" test sets containing heavy noise, where WERR reductions exceeded 10% compared to supervised baselines. The inherent parallelizability of the Transformer architecture allows Emformer to achieve these gains while maintaining superior inference speed, making it highly suitable for large-scale production environments.

Conclusion

Emformer represents a significant step forward in making high-performance Transformer models viable for real-time ASR. While a small "streaming penalty" remains compared to full offline Transformers, the gains in efficiency and accuracy over LSTMs are compelling. Future research will explore the integration of convolutional layers to further optimize model size and close the remaining performance gap with non-streaming architectures.

Summary updated

Abstract

In this paper, we summarize the application of transformer and its streamable variant, Emformer based acoustic model for large scale speech recognition applications. We compare the transformer based acoustic models with their LSTM counterparts on industrial scale tasks. Specifically, we compare Emformer with latency-controlled BLSTM (LCBLSTM) on medium latency tasks and LSTM on low latency tasks. On a low latency voice assistant task, Emformer gets 24% to 26% relative word error rate reductions (WERRs). For medium latency scenarios, comparing with LCBLSTM with similar model size and latency, Emformer gets significant WERR across four languages in video captioning datasets with 2-3 times inference real-time factors reduction.
BibTeX
@inproceedings{wang2021transformer,
  author = {Yongqiang Wang and Yangyang Shi and Frank Zhang and Chunyang Wu and Julian Chan and Ching-Feng Yeh and Alex Xiao},
  title = {{Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414087},
}
Figure