Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization

Zhengkun Tian, Hongyu Xiang, Min Li, Feifei Lin, Ke Ding, Guanglu Wan

TL;DR

Researchers at Meituan have developed Peak-First Regularization (PFR), a simple yet effective method to reduce the "peak latency" in Connectionist Temporal Classification (CTC) speech recognition models. By using frame-wise knowledge distillation to encourage the model to predict tokens slightly earlier, they reduced latency by 100–200 milliseconds with virtually no loss in accuracy.

What Problem Does This Paper Address?

In Automatic Speech Recognition (ASR), CTC models are favorites because they are fast and simple. However, they suffer from a specific type of delay called peak latency. This happens because the model often waits for more acoustic context before "committing" to a character, resulting in a probability peak that appears later than the actual speech signal.

While several methods exist to force the model to predict earlier (like FastEmit or constrained alignment), they are often "heavy-lift" solutions. They usually require modifying the complex forward-backward algorithm, changing how gradients are calculated, or relying on external "ground truth" alignments from other pre-trained models. The authors sought a "plug-and-play" regularization term that could achieve the same goal without touching the core CTC machinery.

Methods

The authors introduce Peak-First Regularization (PFR). The intuition is elegant: if you want the model to predict a token at time $t$ instead of time $t+1$, why not just tell the model at time $t$ to mimic what it would have predicted at $t+1$?

Key components include:

  • Frame-wise Knowledge Distillation: The model treats its own future prediction (frame $t+1$) as a "teacher" for its current prediction (frame $t$).
  • KL Divergence: The PFR loss is calculated as the Kullback–Leibler divergence between the probability distributions of adjacent frames.
  • Temperature Smoothing: They use a temperature coefficient ($\tau = 10.0$) to smooth the probability distributions, preventing the model from becoming overconfident in a single token too early, which could destabilize training.
  • Joint Optimization: The final loss is a weighted sum: $L = L_{CTC} + \lambda L_{PFR}$. By adjusting the hyperparameter $\lambda$, developers can trade off between aggressive latency reduction and recognition accuracy.

Datasets

The experiments were conducted on AISHELL-1, a well-known public Chinese Mandarin speech corpus.

  • Training: ~150 hours (120,098 utterances).
  • Development: ~20 hours.
  • Testing: ~10 hours.
  • Features: 83-dimensional FBANK with pitch features.

Key Insights and Ideas

  • The "Shift-Left" Intuition: The authors visualized the CTC alignment path space and noted that lower-latency paths are physically located further to the left on the time axis. PFR acts as a force "pulling" the probability distribution to the left.
  • Gradient Boosting: Mathematically, the authors show that PFR increases the gradient of the current frame based on the probability of the next frame. If the next frame is likely to be a specific character, the current frame is strongly encouraged to "jump the gun" and predict it now.
  • Implicit Knowledge Distillation: Interestingly, the authors found that PFR sometimes improves accuracy. They hypothesize that because the right (future) frame has access to more acoustic context, it acts as a more informed teacher for the left (past) frame.
  • Streaming vs. Non-Streaming: The paper highlights that non-streaming models often have "negative" latency (predicting before the phoneme ends) because they see the whole sentence, while streaming models have positive latency due to their limited lookahead (510ms in this study).

Key Results

The method was tested on both streaming and non-streaming Transformer-based CTC models:

  • Non-Streaming Models: With a weight of $\lambda=0.9$, the average peak latency was reduced by 216.85ms. The Character Error Rate (CER) actually improved slightly from 5.85% to 5.71%.
  • Streaming Models: With a weight of $\lambda=5.0$, the average peak latency dropped by 178.51ms. The CER saw a negligible increase (from 7.03% to 7.22%), which is a small price to pay for nearly 200ms of saved time in a real-time application.
  • Consistency: Across various weights, the "peaks" in the probability distribution clearly shifted earlier in time without losing their sharpness.

Limitations and Future Work

  • Hyperparameter Tuning: The optimal weight $\lambda$ varies significantly between streaming and non-streaming models. Finding the "sweet spot" requires manual tuning to prevent the training from collapsing.
  • Language Specificity: The study focused exclusively on Mandarin Chinese. Since Mandarin is a tonal, character-based language, the dynamics of "peaks" might differ in alphabetic languages like English or German.
  • Future Directions: The authors plan to verify the method on other language datasets and explore even more aggressive ways to reduce latency without sacrificing the simplicity of the CTC framework.

Summary updated

Abstract

The CTC model has been widely applied to many application scenarios because of its simple structure, excellent performance, and fast inference speed. There are many peaks in the probability distribution predicted by the CTC models, and each peak represents a non-blank token. The recognition latency of CTC models can be reduced by encouraging the model to predict peaks earlier. Existing methods to reduce latency require modifying the transition relationship between tokens in the forward-backward algorithm, and the gradient calculation. Some of these methods even depend on the forced alignment results provided by other pretrained models. The above methods are complex to implement. To reduce the peak latency, we propose a simple and novel method named peak-first regularization, which utilizes a frame-wise knowledge distillation function to force the probability distribution of the CTC model to shift left along the time axis instead of directly modifying the calculation process of CTC loss and gradients. All the experiments are conducted on a Chinese Mandarin dataset AISHELL-1. We have verified the effectiveness of the proposed regularization on both streaming and non-streaming CTC models respectively. The results show that the proposed method can reduce the average peak latency by about 100 to 200 milliseconds with almost no degradation of recognition accuracy.
BibTeX
@inproceedings{tian2023peak,
  author = {Zhengkun Tian and Hongyu Xiang and Min Li and Feifei Lin and Ke Ding and Guanglu Wan},
  title = {{Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=XUTY6ZR5LO},
}

Loading related content...

Figure