Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization

Zhengkun Tian, Hongyu Xiang, Min Li, Feifei Lin, Ke Ding, Guanglu Wan

Abstract

The CTC model has been widely applied to many application scenarios because of its simple structure, excellent performance, and fast inference speed. There are many peaks in the probability distribution predicted by the CTC models, and each peak represents a non-blank token. The recognition latency of CTC models can be reduced by encouraging the model to predict peaks earlier. Existing methods to reduce latency require modifying the transition relationship between tokens in the forward-backward algorithm, and the gradient calculation. Some of these methods even depend on the forced alignment results provided by other pretrained models. The above methods are complex to implement. To reduce the peak latency, we propose a simple and novel method named peak-first regularization, which utilizes a frame-wise knowledge distillation function to force the probability distribution of the CTC model to shift left along the time axis instead of directly modifying the calculation process of CTC loss and gradients. All the experiments are conducted on a Chinese Mandarin dataset AISHELL-1. We have verified the effectiveness of the proposed regularization on both streaming and non-streaming CTC models respectively. The results show that the proposed method can reduce the average peak latency by about 100 to 200 milliseconds with almost no degradation of recognition accuracy.

Introduction

In Automatic Speech Recognition (ASR), Connectionist Temporal Classification (CTC) models are widely used due to their simplicity and inference efficiency. However, they often exhibit "peak latency," where the model waits for significant acoustic context before outputting a non-blank token. This delay results in probability peaks appearing later than the corresponding speech signal.

Existing solutions like FastEmit or constrained alignment often involve complex modifications to the forward-backward algorithm or require external alignments. Researchers at Meituan have developed Peak-First Regularization (PFR), a simple yet effective method to address this by encouraging the model to predict tokens slightly earlier through frame-wise knowledge distillation.

Peak-First Regularization (PFR)

The proposed PFR is a "plug-and-play" method that encourages the model to predict tokens earlier without altering the core CTC machinery. The core idea is to treat the model's future prediction (at time $t+1$) as a "teacher" for its current prediction (at time $t$).

The CTC alignment path space and the calculation of Peak-First Regularization.

As shown in Figure 1, PFR effectively "pulls" the probability distribution to the left on the time axis. This is achieved through frame-wise knowledge distillation using Kullback–Leibler (KL) divergence between adjacent frames. A temperature coefficient ($\tau = 10.0$) is applied to smooth the distributions, preventing instability during training. The final loss function is a weighted sum of the standard CTC loss and the PFR loss: $L = L_{CTC} + \lambda L_{PFR}$.

Experimental Setup and Results

Experiments were conducted on the AISHELL-1 dataset, a 150-hour Chinese Mandarin corpus. The method was evaluated on both streaming and non-streaming Transformer-based CTC models.

Non-Streaming Models

For non-streaming models, which have access to the full utterance, PFR with $\lambda=0.9$ reduced average peak latency by 216.85ms. Notably, the Character Error Rate (CER) improved slightly from 5.85% to 5.71%, suggesting that the future frames act as informative teachers for past frames.

Streaming Models

Streaming models, constrained by a 510ms lookahead, also benefited significantly. With $\lambda=5.0$, the average peak latency was reduced by 178.51ms with a negligible CER increase from 7.03% to 7.22%.

Posterior probability peaks for different streaming models, showing the leftward shift with PFR.

Figure 3 illustrates how the peaks in the probability distribution shift earlier in time as the PFR weight increases, while maintaining their sharpness and clarity.

Conclusion

Peak-First Regularization offers a simple, effective way to reduce CTC latency by 100–200ms with minimal impact on accuracy. While hyperparameter tuning for $\lambda$ is necessary to prevent training collapse, the method's simplicity and performance make it a compelling alternative to more complex alignment-based techniques. Future work will explore its applicability to other languages and more aggressive latency reduction strategies.

Summary updated

Abstract

The CTC model has been widely applied to many application scenarios because of its simple structure, excellent performance, and fast inference speed. There are many peaks in the probability distribution predicted by the CTC models, and each peak represents a non-blank token. The recognition latency of CTC models can be reduced by encouraging the model to predict peaks earlier. Existing methods to reduce latency require modifying the transition relationship between tokens in the forward-backward algorithm, and the gradient calculation. Some of these methods even depend on the forced alignment results provided by other pretrained models. The above methods are complex to implement. To reduce the peak latency, we propose a simple and novel method named peak-first regularization, which utilizes a frame-wise knowledge distillation function to force the probability distribution of the CTC model to shift left along the time axis instead of directly modifying the calculation process of CTC loss and gradients. All the experiments are conducted on a Chinese Mandarin dataset AISHELL-1. We have verified the effectiveness of the proposed regularization on both streaming and non-streaming CTC models respectively. The results show that the proposed method can reduce the average peak latency by about 100 to 200 milliseconds with almost no degradation of recognition accuracy.
BibTeX
@inproceedings{tian2023peak,
  author = {Zhengkun Tian and Hongyu Xiang and Min Li and Feifei Lin and Ke Ding and Guanglu Wan},
  title = {{Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=XUTY6ZR5LO},
}
Figure