Delay-penalized transducer for low-latency streaming ASR

Wei Kang, Zengwei Yao, Fangjun Kuang, Liyong Guo, Xiaoyu Yang, Long Lin, Piotr Zelasko, Daniel Povey

Abstract

In streaming automatic speech recognition (ASR), it is desirable to reduce latency as much as possible while having minimum impact on recognition accuracy. Although a few existing methods are able to achieve this goal, they are difficult to implement due to their dependency on external alignments. In this paper, we propose a simple way to penalize symbol delay in transducer model, so that we can balance the trade-off between symbol delay and accuracy for streaming models without external alignments. Specifically, our method adds a small constant times (T/2 - t), where T is the number of frames and t is the current frame, to all the non-blank log-probabilities (after normalization) that are fed into the two dimensional transducer recursion. For both streaming Conformer models and unidirectional long short-term memory (LSTM) models, experimental results show that it can significantly reduce the symbol delay with an acceptable performance degradation. Our method achieves similar delay-accuracy trade-off to the previously published FastEmit, but we believe our method is preferable because it has a better justification: it is equivalent to penalizing the average symbol delay. Our work is open-sourced and publicly available.

Introduction

In streaming Automatic Speech Recognition (ASR)—the technology powering real-time captions and voice assistants—latency is a critical performance metric. Ideally, text should appear the moment a user speaks. The industry-standard architecture for this task is the Transducer (or RNN-T) model. However, Transducers often exhibit a "procrastination" behavior: to maximize accuracy, the model waits for additional future audio context before committing to a word. This results in "symbol delay," where the text lags behind the speech.

Existing solutions to this problem are often complex. Some require "forced alignments" from external models to supervise the timing of word emissions, which complicates the training pipeline. Others, like the popular FastEmit, manipulate gradients directly during backpropagation. While effective, FastEmit lacks a rigorous mathematical foundation for its behavior. This paper introduces a simpler, mathematically justified modification to the Transducer objective function to penalize delay directly.

Methodology: Penalizing Symbol Delay

The proposed method modifies the log-probabilities of non-blank symbols (actual words or phonemes) during training. Instead of altering the training algorithm itself, we add a simple penalty term to the normalized log-probabilities fed into the transducer recursion:

$$y'(t, u) = y(t, u) + \lambda \times \left(\frac{T}{2} - t\right)$$

Where $t$ is the current frame index, $T$ is the total number of frames, and $\lambda$ is a tunable hyperparameter. By adding this term, frames occurring earlier in time ($t$ is small) receive a probability "bonus," while frames occurring later receive a "penalty."

Lattice visualization of the delay penalty applied to non-blank transitions. Red paths indicate early emissions receiving bonuses, while blue paths indicate late emissions receiving penalties.

This modification is approximately equivalent to adding a regularization term to the loss function that minimizes the weighted average symbol delay. It essentially incentivizes the model to be decisive earlier in the utterance.

Experimental Evaluation

The researchers evaluated the method using the LibriSpeech dataset (1,000 hours of English speech). To ensure the method's generalizability, it was tested on two distinct architectures:

  1. Streaming Conformer: A modern architecture that processes audio in chunks.
  2. Unidirectional LSTM: A classic recurrent architecture.

Standard data augmentations, including SpecAugment and noise injection, were used to ensure model robustness.

Results and Discussion

The proposed delay-penalty method was compared against FastEmit, the current state-of-the-art for latency reduction. The results demonstrate that the method effectively balances the trade-off between Word Error Rate (WER) and latency metrics like Mean Alignment Delay (MAD) and Mean Emission Delay (MED).

Comparison of MAD-WER and MED-WER trade-offs between FastEmit and the proposed delay-penalty method, showing comparable performance.

Key findings include:

  • Significant Latency Reduction: For a Conformer model, the MAD dropped from 373ms (no penalty) to 102ms with only a minor increase in WER (from 3.4% to 3.7%).
  • Negative Latency: With high penalty values ($\lambda$), the MAD can actually become negative. This suggests the model is regularized so aggressively that it predicts and emits symbols before the ground-truth alignment.
  • Consistency: The method performed consistently across both Conformer and LSTM architectures, proving it is a general-purpose tool for Transducer-based ASR.

Conclusion

The delay-penalized transducer offers a simple, effective, and mathematically grounded approach to reducing latency in streaming ASR. By modifying the log-probabilities directly, it avoids the complexity of external alignments while achieving performance parity with state-of-the-art heuristic methods. The implementation is open-sourced as part of the k2-fsa/k2 project, providing a practical tool for the ASR community.

Summary updated

Abstract

In streaming automatic speech recognition (ASR), it is desirable to reduce latency as much as possible while having minimum impact on recognition accuracy. Although a few existing methods are able to achieve this goal, they are difficult to implement due to their dependency on external alignments. In this paper, we propose a simple way to penalize symbol delay in transducer model, so that we can balance the trade-off between symbol delay and accuracy for streaming models without external alignments. Specifically, our method adds a small constant times (T/2 - t), where T is the number of frames and t is the current frame, to all the non-blank log-probabilities (after normalization) that are fed into the two dimensional transducer recursion. For both streaming Conformer models and unidirectional long short-term memory (LSTM) models, experimental results show that it can significantly reduce the symbol delay with an acceptable performance degradation. Our method achieves similar delay-accuracy trade-off to the previously published FastEmit, but we believe our method is preferable because it has a better justification: it is equivalent to penalizing the average symbol delay. Our work is open-sourced and publicly available (https://github.com/k2-fsa/k2).
BibTeX
@inproceedings{kang2023delay,
  author = {Wei Kang and Zengwei Yao and Fangjun Kuang and Liyong Guo and Xiaoyu Yang and Long Lin and Piotr Zelasko and Daniel Povey},
  title = {{Delay-penalized transducer for low-latency streaming ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=SPfu6j7oqG},
}
Figure