Factorized Blank Thresholding for Improved Runtime Efficiency of Neural Transducers

Manh Duc Le, Frank Seide, Yuhao Wang, Li Yang, Kjell Schubert, Ozlem Kalinli, Michael L. SeltzerICASSP 2022

TL;DR

Researchers at Meta have developed a "Factorized Blank Thresholding" technique for RNN-Transducer (RNN-T) models that slashes on-device power consumption by up to 53% and speeds up decoding by 30%. By separating the calculation of "blank" symbols from actual text tokens and skipping the expensive parts when a blank is likely, they achieve these gains with zero loss in recognition accuracy.

What Problem Does This Paper Address?

On-device Automatic Speech Recognition (ASR) is a balancing act between accuracy, latency, and battery life. The RNN-Transducer (RNN-T) has become the industry standard for this task because it's compact and handles streaming audio well.

While most research focuses on optimizing the "Encoder" (the part that processes audio) or the "Predictor" (the part that predicts the next word), this paper shines a light on the Joiner. The Joiner combines the outputs of the encoder and predictor to decide which character or word to output next.

Even though the Joiner is mathematically simple (often just a single layer), it is a massive efficiency bottleneck because it is executed 80x more frequently than the encoder during decoding. In mobile hardware, the constant "weight loading" (moving data from main memory to the processor) required for these frequent Joiner calls drains the battery rapidly.

Methods

The authors introduce Factorized Blank Thresholding, which relies on two key components:

  1. HAT-style Factorization: Borrowing from the Hybrid Autoregressive Transducer (HAT) architecture, they split the Joiner into two separate paths: one that calculates the probability of a "blank" symbol (indicating no new text at this millisecond) and one that calculates the probabilities of all actual vocabulary tokens (the "non-blank" path).
  2. Conditional Execution (The Threshold): RNN-T models are notoriously "spiky"—they output blanks for the vast majority of time steps. The authors propose calculating the blank probability first. If this probability is above a certain threshold (meaning the model is very sure there is no new text), the system skips the much more computationally expensive non-blank path entirely.

Crucially, because the "blank" Joiner is tiny (projecting to a single value), its weights can stay in the high-speed local cache (SRAM), while the much larger "non-blank" Joiner weights only need to be fetched from the power-hungry main memory (DDR) when absolutely necessary.

Datasets

The researchers evaluated their approach on two major benchmarks:

  • LibriSpeech: A standard public dataset of 960 hours of read English speech.
  • Large-Scale In-House Data: A massive dataset of 83 million hand-transcribed utterances (approx. 145,000 hours) collected from Meta Portal and Quest devices to test real-world robustness.

Key Insights and Ideas

  • The Joiner is a Hidden Power Hog: While the Encoder has more parameters, the Joiner’s high invocation frequency makes it the dominant consumer of memory bandwidth.
  • Exploiting "Spikiness": The authors found that for roughly two-thirds of Joiner calls, the model is so certain of a blank that the non-blank computation is redundant.
  • Memory Bandwidth is the Real Enemy: On mobile chips, the energy cost of moving data (loading weights) is often much higher than the cost of the actual math. By reducing the number of times the large non-blank weight matrix is loaded, they achieve massive power savings.
  • Accuracy is Preserved: Because the model is already very confident when it predicts a blank, skipping the alternative doesn't change the final transcript.

Key Results

The results were consistent across both public and private datasets:

  • Speed: Achieved a 26-30% improvement in overall decoding speed (RTF).
  • Power: Reduced on-device power consumption by 43-53%.
  • Efficiency: Successfully skipped roughly 66% of non-blank Joiner invocations.
  • Accuracy: There was zero Word Error Rate (WER) degradation on LibriSpeech and only negligible impact on out-of-domain in-house data when using a conservative threshold.

Limitations and Future Work

The main limitation identified is that a fixed threshold might not be optimal for all scenarios. For example, on "out-of-domain" data (audio that sounds very different from the training data), the model is less confident, and a fixed threshold might cause it to skip computations it actually needs, leading to slight accuracy drops.

The authors suggest two future directions:

  1. Dynamic Thresholding: Adjusting the threshold on the fly based on the model's confidence.
  2. Trainable Thresholds: Integrating the thresholding logic directly into the model's training process so it learns exactly when it can safely "skip" the hard work.

Summary updated

Abstract

We show how factoring the RNN-T's output distribution can significantly reduce the computation cost and power consumption for on-device ASR inference with no loss in accuracy. With the rise in popularity of neural-transducer type models like the RNN-T for on-device ASR, optimizing RNN-T's runtime efficiency is of great interest. While previous work has primarily focused on the optimization of RNN-T's acoustic encoder and predictor, this paper focuses the attention on the joiner. We show that despite being only a small part of RNN-T, the joiner has a large impact on the overall model's runtime efficiency. We propose to utilize HAT-style joiner factorization for the purpose of skipping the more expensive non-blank computation when the blank probability exceeds a certain threshold. Since the blank probability can be computed very efficiently and the RNN-T output is dominated by blanks, our proposed method leads to a 26-30% decoding speed-up and 43-53% reduction in on-device power consumption, all the while incurring no accuracy degradation and being relatively simple to implement.
BibTeX
@inproceedings{le2022factorized,
  author = {Manh Duc Le and Frank Seide and Yuhao Wang and Li Yang and Kjell Schubert and Ozlem Kalinli and Michael L. Seltzer},
  title = {{Factorized Blank Thresholding for Improved Runtime Efficiency of Neural Transducers}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
}
Figure