Abstract
There has been increasing interest in unifying streaming and non-streaming automatic speech recognition (ASR) models to reduce development, training, and deployment costs. We present a unified framework that trains a single end-to-end ASR model for both streaming and non-streaming applications, leveraging future context information. We propose to use dynamic right-context through the chunked attention masking in the training of zipformer-based ASR models. We demonstrate that using right-context is more effective in zipformer models compared to other conformer models due to its multi-scale nature. We analyze the effect of varying the number of right-context frames on accuracy and latency of the streaming ASR models. We use Librispeech and large in-house conversational datasets to train different versions of streaming and non-streaming models and evaluate them in a production grade server-client setup across diverse testsets of different domains. The proposed strategy reduces word error by relative 7.9% with a small degradation in user-perceived latency. By adding more right-context frames, we are able to achieve streaming performance close to that of non-streaming models. Our approach also allows flexible control of the latency-accuracy tradeoff according to customers requirements.
Introduction
In automatic speech recognition (ASR), a fundamental trade-off exists between latency and accuracy. Streaming models provide near-instantaneous transcriptions but suffer from lower accuracy because they cannot access future audio context. Conversely, non-streaming (offline) models process the entire audio sequence, achieving higher accuracy at the cost of being unsuitable for real-time use. Historically, these requirements necessitated maintaining two separate models, increasing engineering and computational overhead. This paper introduces a unified Zipformer-based framework that bridges this gap, allowing a single model to serve both low-latency streaming and high-accuracy offline applications through dynamic right-context masking.
Methodology
The proposed framework is built upon the Zipformer architecture, an evolution of the Conformer designed for greater speed and memory efficiency. Two primary innovations drive the unification:
Multi-Scale Temporal Modeling
The Zipformer utilizes a U-Net-like structure to process audio at multiple resolutions, ranging from 50Hz down to 6.25Hz. This multi-scale approach allows the model to integrate temporal information more effectively across different resolutions.
Dynamic Right-Context Masking
To enable a single model to handle varying latency requirements, the researchers employ dynamic right-context masking during training. Instead of a fixed lookahead, the number of "future" frames (right-context) is randomly varied (e.g., 0, 64, 128, or 256 frames) for each training batch. This jittering forces the model to learn robust representations regardless of the available future context. At inference, the amount of right-context can be adjusted to meet specific latency or accuracy targets.
Experimental Setup
The model was evaluated on two major datasets:
- Librispeech (960 hours): A standard academic benchmark for English ASR.
- Large In-house Conversational Dataset (12,468 hours): A diverse collection of telephonic data, synthesized speech, and multiple English accents (Indian, Filipino, Australian, UK) across domains such as banking, healthcare, and retail.
Results and Discussion
The unified Zipformer model demonstrates a significant advantage over traditional Conformer architectures when utilizing right-context.
Performance Gains
In a production-grade server-client environment, the dynamic right-context strategy reduced the Word Error Rate (WER) by a relative 7.9% compared to a baseline streaming model. On the large in-house dataset, the unified model with 64 frames of right-context achieved a WER of 8.31%, nearly matching the 8.26% WER of a dedicated non-streaming model.
Latency and Throughput
The impact on user-perceived latency was minimal. For instance, at a concurrency of 300 calls, latency increased from 2.45s (zero right-context) to 3.24s (64 frames), a small trade-off for the substantial accuracy gains. The model also maintained high throughput, processing hundreds of concurrent streams on a single Nvidia A10G GPU.
Conclusion
This work demonstrates that the multi-scale nature of Zipformers makes them uniquely suited for unified ASR. By training with variable right-context, a single model can effectively serve both streaming and offline use cases. Future work will focus on further optimizing the latency-accuracy "sweet spot" and improving domain adaptation across highly varied datasets.
Summary updated