Unifying Streaming and Non-streaming Zipformer-based ASR

Bidisha Sharma, Karthik Pandia Durai, Shankar Venkatesan, Jeena J. Prakash, Shashi Kumar, Malolan Chetlur, Andreas Stolcke

Abstract

There has been increasing interest in unifying streaming and non-streaming automatic speech recognition (ASR) models to reduce development, training, and deployment costs. We present a unified framework that trains a single end-to-end ASR model for both streaming and non-streaming applications, leveraging future context information. We propose to use dynamic right-context through the chunked attention masking in the training of zipformer-based ASR models. We demonstrate that using right-context is more effective in zipformer models compared to other conformer models due to its multi-scale nature. We analyze the effect of varying the number of right-context frames on accuracy and latency of the streaming ASR models. We use Librispeech and large in-house conversational datasets to train different versions of streaming and non-streaming models and evaluate them in a production grade server-client setup across diverse testsets of different domains. The proposed strategy reduces word error by relative 7.9% with a small degradation in user-perceived latency. By adding more right-context frames, we are able to achieve streaming performance close to that of non-streaming models. Our approach also allows flexible control of the latency-accuracy tradeoff according to customers requirements.

Introduction

In automatic speech recognition (ASR), a fundamental trade-off exists between latency and accuracy. Streaming models provide near-instantaneous transcriptions but suffer from lower accuracy because they cannot access future audio context. Conversely, non-streaming (offline) models process the entire audio sequence, achieving higher accuracy at the cost of being unsuitable for real-time use. Historically, these requirements necessitated maintaining two separate models, increasing engineering and computational overhead. This paper introduces a unified Zipformer-based framework that bridges this gap, allowing a single model to serve both low-latency streaming and high-accuracy offline applications through dynamic right-context masking.

Methodology

The proposed framework is built upon the Zipformer architecture, an evolution of the Conformer designed for greater speed and memory efficiency. Two primary innovations drive the unification:

Multi-Scale Temporal Modeling

The Zipformer utilizes a U-Net-like structure to process audio at multiple resolutions, ranging from 50Hz down to 6.25Hz. This multi-scale approach allows the model to integrate temporal information more effectively across different resolutions.

The Zipformer architecture utilizes a multi-scale encoder block to process audio at varying temporal resolutions.

Dynamic Right-Context Masking

To enable a single model to handle varying latency requirements, the researchers employ dynamic right-context masking during training. Instead of a fixed lookahead, the number of "future" frames (right-context) is randomly varied (e.g., 0, 64, 128, or 256 frames) for each training batch. This jittering forces the model to learn robust representations regardless of the available future context. At inference, the amount of right-context can be adjusted to meet specific latency or accuracy targets.

Experimental Setup

The model was evaluated on two major datasets:

  1. Librispeech (960 hours): A standard academic benchmark for English ASR.
  2. Large In-house Conversational Dataset (12,468 hours): A diverse collection of telephonic data, synthesized speech, and multiple English accents (Indian, Filipino, Australian, UK) across domains such as banking, healthcare, and retail.

Results and Discussion

The unified Zipformer model demonstrates a significant advantage over traditional Conformer architectures when utilizing right-context.

Comparison of WER between Conformer and Zipformer models as a function of right-context frames. Zipformer shows a more pronounced improvement with increased context.

Performance Gains

In a production-grade server-client environment, the dynamic right-context strategy reduced the Word Error Rate (WER) by a relative 7.9% compared to a baseline streaming model. On the large in-house dataset, the unified model with 64 frames of right-context achieved a WER of 8.31%, nearly matching the 8.26% WER of a dedicated non-streaming model.

Latency and Throughput

The impact on user-perceived latency was minimal. For instance, at a concurrency of 300 calls, latency increased from 2.45s (zero right-context) to 3.24s (64 frames), a small trade-off for the substantial accuracy gains. The model also maintained high throughput, processing hundreds of concurrent streams on a single Nvidia A10G GPU.

Conclusion

This work demonstrates that the multi-scale nature of Zipformers makes them uniquely suited for unified ASR. By training with variable right-context, a single model can effectively serve both streaming and offline use cases. Future work will focus on further optimizing the latency-accuracy "sweet spot" and improving domain adaptation across highly varied datasets.

Summary updated

Abstract

There has been increasing interest in unifying streaming and non-streaming automatic speech recognition (ASR) models to reduce development, training, and deployment costs. We present a unified framework that trains a single end-to-end ASR model for both streaming and non-streaming applications, leveraging future context information. We propose to use dynamic right-context through the chunked attention masking in the training of zipformer-based ASR models. We demonstrate that using right-context is more effective in zipformer models compared to other conformer models due to its multi-scale nature. We analyze the effect of varying the number of right-context frames on accuracy and latency of the streaming ASR models. We use Librispeech and large in-house conversational datasets to train different versions of streaming and non-streaming models and evaluate them in a production grade server-client setup across diverse testsets of different domains. The proposed strategy reduces word error by relative 7.9\% with a small degradation in user-perceived latency. By adding more right-context frames, we are able to achieve streaming performance close to that of non-streaming models. Our approach also allows flexible control of the latency-accuracy tradeoff according to customers requirements.
BibTeX
@inproceedings{sharma2025unifying,
  author = {Bidisha Sharma and Karthik Pandia Durai and Shankar Venkatesan and Jeena J. Prakash and Shashi Kumar and Malolan Chetlur and Andreas Stolcke},
  title = {{Unifying Streaming and Non-streaming Zipformer-based ASR}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=i6fHSY3g7i},
}
Figure