Run-and-Back Stitch Search: Novel Block Synchronous Decoding For Streaming Encoder-Decoder ASR

Emiru Tsunoo, Chaitanya Narisetty, Michael Hentschel, Yosuke Kashiwagi, Shinji Watanabe

Abstract

Streaming Automatic Speech Recognition (ASR) requires low-latency processing of audio as it is captured. While encoder-decoder models offer high accuracy, they face challenges in "endpointing"—determining when to stop decoding a current audio block. This paper presents Run-and-Back Stitch (RABS) search, a novel decoding algorithm that leverages existing model signals to manage block synchronization. RABS combines proactive endpoint prediction (Running Stitch) with a corrective mechanism for over-decoding (Back Stitch). Experimental results on Librispeech, AISHELL-1, and CSJ demonstrate that RABS significantly reduces tail latency (EP90) by up to 45% without compromising accuracy or requiring additional model training.

Introduction

In interactive ASR applications, streaming is essential for responsiveness. Encoder-decoder (Enc-Dec) architectures typically use block-synchronous decoding, where the decoder waits for the encoder to process a chunk of audio. A critical bottleneck in this process is the endpointing problem: the decoder must decide whether to continue generating tokens or wait for more audio data. Over-decoding leads to "hallucinations" or repetitions, which increase latency and degrade Word Error Rate (WER). Conversely, premature stopping misses critical speech information.

Methodology: Run-and-Back Stitch (RABS)

The proposed RABS search addresses endpointing through two integrated strategies that utilize internal model signals: the CTC posterior and Source-Target (ST) attention weights.

Running Stitch (Endpoint Prediction)

The Running Stitch mechanism proactively predicts the end of a speech segment within the current audio block. By analyzing the CTC posterior—which maps character probabilities over time—and the ST attention weights, the system calculates the expected number of tokens remaining. This allows the decoder to pause at the optimal moment, preventing excessive decoding into silence or future audio frames.

Figure 3: Mechanism of endpoint prediction using CTC posteriors and attention weights.

Back Stitch (Post-determination and Correction)

To handle instances where the prediction is inaccurate, the Back Stitch mechanism acts as a safety net. It monitors the monotonicity of attention weights. A "backward jump" in attention—where the model focuses on previously processed audio frames—is a mathematical indicator of over-decoding. RABS employs a probability-based formula to detect these jumps, discards the erroneous hypotheses, and restarts decoding once sufficient new audio data is available. Crucially, this method works with standard CTC/Attention architectures and does not require any additional training.

Experimental Evaluation and Results

The researchers evaluated RABS across three datasets representing different linguistic structures and token granularities: Librispeech (English subwords), AISHELL-1 (Mandarin characters), and the Corpus of Spontaneous Japanese (CSJ, Japanese characters). RABS demonstrated substantial improvements in latency, particularly in the 90th percentile (EP90), which represents the "worst-case" user experience.

Figure 6: Performance comparison of RABS against baseline methods across various model configurations and beam sizes.

Key findings include:

  • Latency Reduction: On Librispeech, RABS reduced EP90 latency from 1487 ms to 821 ms for a regular model, and from 857 ms to 354 ms for a small model, compared to standard block-synchronous decoding (BS-Dec).
  • Efficiency: The "average last decoding steps"—a metric for redundant work performed after the audio ends—dropped from 12.28 to 1.52 in large HuBERT-based models.
  • Robustness: The method proved effective across all three tested languages, confirming its generalizability to different tokenization schemes.

Conclusion

Run-and-Back Stitch search provides an elegant, training-free solution to the endpointing problem in streaming Enc-Dec ASR. By leveraging existing CTC and attention signals, it enables aggressive decoding with a robust corrective mechanism, drastically reducing tail latency while maintaining state-of-the-art accuracy.

Summary updated

BibTeX
@inproceedings{tsunoo2022run,
  author = {Emiru Tsunoo and Chaitanya Narisetty and Michael Hentschel and Yosuke Kashiwagi and Shinji Watanabe},
  title = {{Run-and-Back Stitch Search: Novel Block Synchronous Decoding For Streaming Encoder-Decoder ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747800},
}
Figure