Abstract
In this paper, we summarize the application of transformer and its streamable variant, Emformer based acoustic model for large scale speech recognition applications. We compare the transformer based acoustic models with their LSTM counterparts on industrial scale tasks. Specifically, we compare Emformer with latency-controlled BLSTM (LCBLSTM) on medium latency tasks and LSTM on low latency tasks. On a low latency voice assistant task, Emformer gets 24% to 26% relative word error rate reductions (WERRs). For medium latency scenarios, comparing with LCBLSTM with similar model size and latency, Emformer gets significant WERR across four languages in video captioning datasets with 2-3 times inference real-time factors reduction.
Introduction
The field of Automatic Speech Recognition (ASR) faces a persistent challenge: balancing high accuracy with low latency. While standard Transformer models have set new benchmarks for offline transcription due to their non-causal nature, they are often unsuitable for real-time applications like live captioning or voice assistants. Historically, Long Short-Term Memory (LSTM) networks were the industry standard for streaming tasks. However, LSTMs are difficult to parallelize and lack the modeling capacity of Transformers. This paper introduces Emformer (Efficient Memory Transformer), a streamable variant designed to bridge this gap by providing Transformer-level accuracy with the efficiency required for industrial-scale, real-time deployment.
Methods
The core of the Emformer architecture is a block-processing approach that slices audio into segments. To maintain contextual awareness, each segment incorporates a "right context" (look-ahead) and a "left context" (history). A key innovation in Emformer is its efficient caching mechanism. Unlike previous augmented memory transformers that recomputed historical context at every step, Emformer utilizes pre-computed keys and values from previous segments.
As shown in Figure 1, the model uses a memory bank (M) to summarize long-range history. This allows the model to capture distant dependencies without the quadratic computational overhead of standard self-attention. By caching the keys and values of the left context, the model avoids the redundant computation that plagued previous streaming transformer implementations. This architecture supports both traditional hybrid HMM-based systems and modern neural transducer architectures.
Experimental Setup
The researchers evaluated Emformer on massive, industrial-scale datasets, moving far beyond standard academic benchmarks:
- Voice Assistant Task: A 47k-hour dataset (initially 12.5k hours of supervised data augmented with simulated noise and room acoustics).
- Video ASR Task: Spanning four languages (English, Spanish, Vietnamese, and German) with supervised data ranging from 9k to 39k hours.
- Large-Scale Semi-Supervised Learning: A "Giant" set comprising 2.2 million hours of English data to test the model's scaling limits.
Key Results and Insights
Emformer demonstrated substantial improvements over LSTM baselines across various latency requirements. In low-latency voice assistant tasks (~80-120ms), Emformer achieved a relative Word Error Rate Reduction (WERR) of 24% to 26% compared to standard LSTMs.
For medium-latency scenarios such as video captioning, Emformer was compared against Latency-Controlled Bidirectional LSTMs (LCBLSTM). As detailed in Figure 5, Emformer not only improved accuracy but also significantly reduced the Real-Time Factor (RTF), running 2-3 times faster than LCBLSTM across multiple languages.
The results also highlighted the impact of scale. Training on the 2.2 million-hour dataset provided a significant boost, particularly on "extreme" test sets containing heavy noise, where WERR reductions exceeded 10% compared to supervised baselines. The inherent parallelizability of the Transformer architecture allows Emformer to achieve these gains while maintaining superior inference speed, making it highly suitable for large-scale production environments.
Conclusion
Emformer represents a significant step forward in making high-performance Transformer models viable for real-time ASR. While a small "streaming penalty" remains compared to full offline Transformers, the gains in efficiency and accuracy over LSTMs are compelling. Future research will explore the integration of convolutional layers to further optimize model size and close the remaining performance gap with non-streaming architectures.
Summary updated