Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper

Iuliia Thorbecke, Juan Pablo Zuluaga-Gomez, Esaú Villatoro-Tello, Shashi Kumar, Pradeep Rangappa, Sergio Burdisso, Petr Motlícek, Karthik S, Aravind Ganapathiraju

Abstract

The training of automatic speech recognition (ASR) with little to no supervised data remains an open question. In this work, we demonstrate that streaming Transformer-Transducer (TT) models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled (PL) speech from foundational speech models (FSM). This allows training a robust ASR model just in one stage and does not require large data and computational budget compared to the two-step scenario with pre-training and fine-tuning. We perform a comprehensive ablation on different aspects of PL-based streaming TT models such as the impact of (1) shallow fusion of n-gram LMs, (2) contextual biasing with named entities, (3) chunk-wise decoding for low-latency streaming applications, and (4) TT overall performance as the function of the FSM size. Our results demonstrate that TT can be trained from scratch without supervised data, even with very noisy PLs. We validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.

Introduction

Building high-performance Automatic Speech Recognition (ASR) systems traditionally necessitates thousands of hours of human-transcribed audio, a resource that is often unavailable for many languages or specific domains. While foundational models (FSMs) like Whisper offer impressive accuracy, their large parameter counts make them unsuitable for low-latency, real-time streaming applications. Furthermore, adapting these models typically involves a costly two-step process of self-supervised pre-training followed by supervised fine-tuning.

This research explores zero-resource prototyping: the ability to build a lightweight, production-ready streaming Transformer-Transducer (TT) model without any human-labeled data. By using a large FSM as a "teacher" to generate pseudo-labels for unlabeled audio, the authors demonstrate a path toward rapid ASR development on consumer-grade hardware.

Methodology

The proposed framework utilizes sequence-level knowledge distillation to train a student model from a teacher model.

Pseudo-Labeling and Filtering

The pipeline begins by transcribing unlabeled audio using Whisper (via WhisperX). To mitigate Whisper's tendency to "hallucinate" (e.g., repeating unigrams or generating text unrelated to the audio), the authors implemented several heuristic filters:

  • Repetition Check: Utterances with excessive repeated unigrams are discarded.
  • Statistics-based Filtering: Audio-to-text duration ratios and maximum word lengths are used to prune outliers.
  • Text Normalization: Standardizing numbers and removing punctuation to ensure a cleaner training signal.

Model Training and Decoding

A Zipformer (a state-of-the-art Transducer architecture) is trained from scratch using these pseudo-labels. Unlike traditional methods, this is a "one-stage" process that bypasses pre-training. To support streaming, the model employs multi-chunk training.

The two-stage framework: 1) Efficient pseudo-labeling with a foundational model and 2) Transformer-Transducer training and decoding.

During the decoding phase, the system uses Shallow Fusion to integrate an external n-gram language model. Additionally, an Aho-Corasick based biasing list is used to improve the recognition of Named Entities (NEs), which are often poorly captured in low-resource settings.

Experimental Setup

The framework was validated using:

  • CommonVoice-v11: Spanning six languages: English (EN), Catalan (CA), French (FR), German (DE), Spanish (ES), and Italian (IT).
  • Industrial Call-Center Data: Approximately 1,728 hours of unlabeled, noisy agent-user conversations to test domain adaptation.

Results and Discussion

The experiments reveal that Transducer models are remarkably resilient to the noise present in pseudo-labels, provided the data volume is sufficient.

Accuracy and Scaling

In several cases, models trained purely on pseudo-labels rivaled or exceeded those trained on limited supervised data. For instance, in Italian (IT), the pseudo-labeled model achieved a Word Error Rate (WER) of 17.2%, outperforming the supervised baseline of 17.5%. As shown in the scaling analysis, performance improves consistently as the model size increases, and the gap between purely pseudo-labeled and supervised-augmented training narrows significantly for larger models.

Word Error Rate (WER) as a function of model size across six languages, comparing models trained with and without additional supervised data.

Impact of Teacher Size and Heuristics

The quality of the student model is directly correlated with the size of the teacher model (e.g., whisper-large-v3 vs. whisper-tiny). However, even smaller teachers provide a viable starting point. Crucially, the filtering heuristics are essential for convergence, as they remove the "hallucinated" noise that would otherwise degrade the student's performance.

Efficiency

A significant advantage of this approach is its computational efficiency. Training a robust model for a new language can be completed in 1-2 days on a single NVIDIA RTX 3090 GPU, making it a highly accessible method for rapid prototyping.

Conclusion

This work demonstrates that streaming Transformer-Transducers can be effectively trained from scratch using only pseudo-labels from foundational models. By combining smart data filtering with shallow fusion and contextual biasing, it is possible to create high-quality ASR systems for new domains or languages without the need for expensive human transcription. Future work may explore the application of this method to spontaneous speech and a wider variety of linguistic structures, such as tonal languages.

Summary updated

Abstract

The training of automatic speech recognition (ASR) with little to no supervised data remains an open question. In this work, we demonstrate that streaming Transformer-Transducer (TT) models can be trained from scratch in consumer and accessible GPUs in their entirety with pseudo-labeled (PL) speech from foundational speech models (FSM). This allows training a robust ASR model just in one stage and does not require large data and computational budget compared to the two-step scenario with pre-training and fine-tuning. We perform a comprehensive ablation on different aspects of PL-based streaming TT models such as the impact of (1) shallow fusion of n-gram LMs, (2) contextual biasing with named entities, (3) chunk-wise decoding for low-latency streaming applications, and (4) TT overall performance as the function of the FSM size. Our results demonstrate that TT can be trained from scratch without supervised data, even with very noisy PLs. We validate the proposed framework on 6 languages from CommonVoice and propose multiple heuristics to filter out hallucinated PLs.
BibTeX
@inproceedings{thorbecke2024fast,
  author = {Iuliia Thorbecke and Juan Pablo Zuluaga-Gomez and Esaú Villatoro-Tello and Shashi Kumar and Pradeep Rangappa and Sergio Burdisso and Petr Motlícek and Karthik S and Aravind Ganapathiraju},
  title = {{Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=oOX2lUc5Wq},
}
Figure