TL;DR
Deploying separate automatic speech recognition (ASR) models for high-accuracy offline transcription and low-latency streaming is costly and inefficient. This paper introduces a unified Transducer (RNNT) framework that shares a single set of parameters for both modes, using a novel Mode-Consistency Regularization (MCR-RNNT) loss implemented via efficient Triton kernels to bridge the performance gap between offline and streaming decoding.
What Problem Does This Paper Address?
In automatic speech recognition, production systems typically demand two distinct capabilities:
- Offline Decoding: Processing an entire audio file at once to achieve maximum transcription accuracy.
- Streaming Decoding: Processing audio in real-time chunks with minimal delay (low latency).
Building a single "unified" model that excels at both is notoriously difficult. Streaming models are starved of future context, which fundamentally changes how their internal encoders represent audio compared to full-context offline models. When forced to share parameters, the model faces a "mode conflict," often resulting in a sharp drop in accuracy for either the offline or streaming regime—especially at ultra-low latencies (under 0.5 seconds).
Methods
The authors build their unified framework on a Conformer-style RNN-Transducer (RNNT) architecture and introduce three main components to harmonize the two decoding modes:
- Chunk-Limited Attention & Dynamic Chunk Convolution (DCConv): During streaming training, the multi-head attention and convolution layers are restricted using a chunked mask. The model only looks at the past (left context), the current audio chunk, and a small, variable window of the future (right context).
- Dual-Mode (DM) Training: Instead of randomly alternating between training modes, each optimization step processes the same input batch twice—once in offline mode and once in streaming mode—combining their respective RNNT losses.
- Mode-Consistency Regularization (MCR-RNNT): To explicitly close the representation gap, the authors introduce a symmetric Kullback-Leibler (KL) divergence loss. This penalty forces the probability distributions of the raw joint network logits from both the offline and streaming modes to agree with one another.
The Engineering Trick: Fused Triton Kernels
Computing a consistency loss across the full RNNT joint tensor is normally a memory nightmare. The tensor size scales as $[Batch, Time, Text, Vocabulary]$, which quickly exhausts GPU memory. To solve this, the authors wrote a custom fused GPU kernel using Triton. It computes the log-softmax and KL divergence on-the-fly and discards intermediate states, recomputing them during the backward pass. This results in virtually zero memory overhead and negligible computational slowdown.
Datasets
- Training Data: The models were trained on subsets of the public Granary dataset: ~120,000 hours of labeled English speech for the Large (L) model, and ~280,000 hours for the Extra-Large (XL) model. The XL dataset included formatting for punctuation and capitalization (PC).
- Evaluation Benchmark: Models were evaluated using the Open ASR Leaderboard for English, which averages Word Error Rate (WER) across 8 diverse datasets: AMI, Earnings22, Gigaspeech, Librispeech, SPGI, TEDLIUM, and VoxPopuli.
Key Insights and Ideas
- Symmetric Regularization is Crucial: Forcing the streaming mode to mimic the offline mode and vice versa creates a stable Pareto frontier. It prevents the model from biasing toward alignments that are easy to compute offline but impossible to achieve under streaming constraints.
- CTC Consistency Fails for Transducers: The authors initially tried applying consistency regularization to auxiliary Connectionist Temporal Classification (CTC) losses (a common technique in hybrid models). However, this degraded streaming RNNT accuracy because CTC forces rigid, frame-synchronous predictions, which conflicts with the flexible alignment space that an RNNT needs to thrive in low-latency settings.
Key Results
The proposed approach significantly outperforms standard unified training methods, maintaining high offline accuracy while remaining robust at low latencies.
- L-Size Model (~128M parameters): The Unified DM + MCR-RNNT model achieved a 6.63% offline WER and maintained a tight 7.47% WER at 0.56s latency. Standard dual-mode training without the consistency loss degraded to 8.12% WER at the same latency.
- XL-Size Model (~600M parameters): When scaled up, the model achieved a state-of-the-art 5.76% Average WER in offline mode on the Open ASR Leaderboard, outperforming strong open-source baselines like Parakeet-TDT-0.6b. In streaming mode, it retained excellent performance down to a 0.24-second latency budget.
Limitations and Future Work
The primary limitation identified is inference speed during streaming. Currently, the framework recalculates the entire left context at each new chunk step, which introduces redundant computations and slows down decoding.
As a next step, the authors plan to implement a cache-passing mechanism to store and reuse historical encoder representations across chunk boundaries, making the streaming inference pipeline highly efficient for production deployment.
Summary updated