Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding

Haoran Zhou, Xingchen Song, Brendan Fahy, Qiaochu Song, Binbin Zhang, Zhendong Peng, Anshul Wadhawan, Denglin Jiang, Apurv Verma, Vinay Ramesh, Srivas Prasad, Michele M. FranceschiniInterspeech 2025

TL;DR

This paper tackles the challenge of making OpenAI's powerful Whisper ASR model suitable for real-time, streaming applications. They achieve this by adapting Whisper into a "two-pass" decoding system, combining a fast, streaming-friendly component with Whisper's original, more accurate but slower component. A novel "hybrid tokenizer" approach further improves performance, especially when fine-tuning data is limited.

What Problem Does This Paper Address?

OpenAI's Whisper model has revolutionized speech recognition with its impressive accuracy and robustness, trained on a massive dataset. However, its core architecture is designed for processing complete audio segments, making it inherently unsuitable for streaming applications where audio arrives in real-time and needs to be transcribed as it comes in. Existing methods to adapt Whisper for streaming often involve workarounds that don't fully integrate it into a true streaming pipeline or lead to performance degradation. This paper aims to bridge this gap by developing a Whisper-based system that can deliver low-latency, accurate transcriptions for live audio.

Methods

The core of the proposed solution is adapting Whisper into a Unified Two-pass (U2) architecture, a framework designed for both streaming and non-streaming ASR. This involves:

  • U2 Architecture: This architecture consists of an encoder (Whisper's encoder), a Connectionist Temporal Classification (CTC) decoder, and an attention decoder (Whisper's original decoder).
  • Hybrid CTC-Attention Loss: During training, a hybrid loss function combines CTC and attention losses. The CTC branch is trained to produce fast, streaming partial transcripts, while the attention decoder is used for more accurate rescoring.
  • Dynamic Attention Masks: To ensure the encoder can operate in a streaming fashion during training, dynamic attention masks are used, making its representations dependent only on past audio.
  • Hybrid Tokenizer: A key innovation is the use of a hybrid tokenizer. The CTC decoder uses a smaller, more granular token space (the first 8,000 tokens from Whisper's tokenizer) to improve data efficiency and generalization, especially for rare words. The original Whisper attention decoder retains its full, larger token space.
  • Streaming Inference: During inference, the CTC decoder generates initial partial transcripts. Once an endpoint is detected (e.g., silence), the attention decoder rescores these partial transcripts to produce the final, more accurate output. The system leverages a key-value (KV) cache for efficient incremental encoder inference.

Datasets

  • LibriSpeech: Used for completeness and comparison with other approaches.
  • Internally Curated Earnings Call Dataset: This is the primary dataset for fine-tuning and evaluation. It's highlighted for its high-quality, written-form transcripts that include punctuation, capitalization, and formatting, making it a challenging and realistic testbed for ASR systems. This dataset was chosen over others like Earnings-22 due to its size and suitability for the experiments. The training set comprises 5,800 hours of audio, and the test set consists of 10 hours from later calls.

Key Insights and Ideas

  • Whisper's Potential for Streaming: The paper demonstrates that Whisper, despite its non-streaming design, can be effectively adapted for streaming ASR with the right architecture and fine-tuning.
  • The Power of Two-Pass Decoding: The U2 architecture effectively balances the need for speed (CTC decoder) with accuracy (attention decoder), a common strategy in streaming ASR.
  • Hybrid Tokenizer for Data Efficiency: The hybrid tokenizer is a crucial finding. By using a smaller token set for the CTC decoder, the model becomes more robust and generalizes better, especially when fine-tuning data is scarce. This is because the smaller set ensures better coverage of essential linguistic units.
  • Trade-offs in Streaming Parameters: The paper clearly illustrates the trade-offs between chunk size, maximum delay, and accuracy, showing that longer delays generally improve accuracy but increase computational cost.

Key Results

  • Improved WER with Hybrid Tokenizer: The hybrid tokenizer consistently outperformed the single-tokenizer approach across various training set sizes on the earnings call dataset, achieving a WER of 17.30% with 5,800 hours of training data, compared to 17.51% for the single tokenizer. The advantage was more pronounced with smaller training sets.
  • Real-time Performance on CPUs: The fine-tuned Whisper Medium models, with optimizations like 8-bit quantization and KV caching, can achieve real-time performance on CPUs.
  • Chunk Size Impact: Accuracy declines as chunk size decreases, with formatting errors becoming more prevalent. For instance, a 100ms chunk size resulted in a WER of 26.93% without rescoring.
  • Rescoring Benefits: Rescoring with the attention decoder provides a modest but consistent improvement in WER, reducing it from 17.60% to 17.30% for a 1000ms chunk size.
  • Comparison with UFAL Whisper: On the earnings dataset and LibriSpeech test-clean, U2 Whisper generally outperformed UFAL Whisper, especially at smaller chunk sizes. However, UFAL Whisper showed an advantage at larger chunk sizes, potentially due to its closer resemblance to non-streaming models.

Limitations and Future Work

  • Finalize Latency: While U2 Whisper achieves real-time processing, the "finalize latency" (time taken for attention rescoring) remains relatively high for real-time applications. This could be improved with smaller Whisper variants (e.g., Whisper Turbo).
  • Data Scaling: The benefits of the hybrid tokenizer diminish as the amount of fine-tuning data increases.
  • Future Work: The authors suggest further leveraging the linguistic knowledge embedded in the pretrained decoder for even better performance.

Summary updated

BibTeX
@inproceedings{zhou2025adapting,
  author = {Haoran Zhou and Xingchen Song and Brendan Fahy and Qiaochu Song and Binbin Zhang and Zhendong Peng and Anshul Wadhawan and Denglin Jiang and Apurv Verma and Vinay Ramesh and Srivas Prasad and Michele M. Franceschini},
  title = {{Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure