TL;DR
This paper introduces a unified ASR framework that trains a single Zipformer model for both streaming and non-streaming speech recognition. By incorporating dynamic "right-context" (future information) during training, they achieve significant word error rate (WER) reductions in streaming models, bringing them closer to the performance of non-streaming models with only a minor increase in latency. This offers a more efficient development and deployment pipeline for ASR systems.
What Problem Does This Paper Address?
Traditionally, Automatic Speech Recognition (ASR) systems have been developed as separate models for "streaming" (real-time, processing audio as it arrives) and "non-streaming" (offline, processing the entire audio file at once) use cases. This duality leads to increased development, training, and maintenance costs, as two distinct model architectures and training pipelines are required. The core challenge is to bridge the performance gap between streaming and non-streaming models, which often arises because non-streaming models can leverage future audio context (right-context) that is unavailable to causal streaming models. This paper aims to unify these two paradigms by enabling a single model to perform well in both scenarios, thereby reducing computational overhead and simplifying deployment.
Methods
The authors propose a novel approach to incorporate "right-context" into the training of Zipformer-based ASR models. Zipformer, an efficient transformer variant for ASR, typically uses chunked attention masking that is causal (only looks at past information). The key innovation here is the introduction of dynamic right-context attention masking during training. This means that for different mini-batches during training, the model is exposed to varying amounts of future audio frames.
Specifically, they modify the attention masking mechanism within the Zipformer architecture. While conventional streaming models use only left-context (past information), this work allows for a variable number of right-context frames to be included. This dynamic approach during training allows the model to learn how to effectively utilize future information, even when deployed in a streaming setting where such information is limited. During inference, the number of right-context frames can be flexibly controlled to balance accuracy and latency. The authors hypothesize that Zipformer's multi-scale nature makes it particularly well-suited for this right-context integration compared to other conformer models.
Datasets
The experiments were conducted on two main datasets:
- LibriSpeech: A standard open-source dataset for ASR research, comprising 960 hours of training data and test-clean/test-other partitions for evaluation.
- Large in-house conversational data: A substantial dataset totaling 12,468 hours, derived from a combination of open-source, in-house conversational, and simulated telephonic datasets. This dataset is more representative of real-world production scenarios and includes diverse domains and accents (e.g., Indian, Filipino, Australian, UK English).
Key Insights and Ideas
- Right-context is crucial for bridging the streaming-non-streaming gap: The experiments clearly demonstrate that even when a model is not explicitly trained with right-context, providing it during inference significantly improves its performance, bringing it closer to non-streaming models.
- Zipformer's architecture benefits from right-context: The authors found that Zipformer models show a more pronounced improvement in WER with the addition of right-context compared to conventional conformer models, suggesting its multi-scale temporal processing is advantageous.
- Dynamic right-context training is effective: Training with a variable number of right-context frames per batch (e.g., randomly selecting from {0, 64, 128, 256}) leads to better performance than using fixed right-context frames during training.
- Flexible latency-accuracy trade-off: The proposed method allows for fine-grained control over the accuracy-latency trade-off during inference by adjusting the number of decoding right-context frames, catering to specific application requirements.
Key Results
- Word Error Rate (WER) Reduction: The proposed strategy reduced word error by 7.9% relative to baseline streaming models.
- Performance Convergence: By increasing the number of decoding right-context frames, streaming models achieved performance very close to their non-streaming counterparts. For instance, on Librispeech, the baseline streaming model's WER improved from 3.33% to 2.81% (test-clean) and 8.90% to 7.36% (test-other) with 256 right-context frames during decoding.
- Latency Impact: A small degradation in user-perceived latency was observed with the introduction of right-context, but this was deemed acceptable given the accuracy gains.
- Server-Client Evaluation: In a production-like server-client setup, the unified streaming model with right-context showed improved WER compared to the baseline streaming model, approaching the performance of the non-streaming model.
Limitations and Future Work
- Latency vs. Accuracy Trade-off: While the paper demonstrates a flexible trade-off, achieving performance identical to non-streaming models might still require a slight increase in latency. The authors acknowledge that further optimization might be needed for latency-critical applications.
- Future Work: The paper suggests that this unified approach can be extended to other transformer-based ASR architectures. Further exploration into optimal strategies for dynamically selecting right-context during training and inference could also be beneficial. The authors also implicitly suggest that this unified model can be deployed in resource-constrained environments, reducing the need for separate streaming and non-streaming models.
Summary updated