TL;DR
Researchers at Meta have developed a "Factorized Blank Thresholding" technique for RNN-Transducer (RNN-T) models that slashes on-device power consumption by up to 53% and speeds up decoding by 30%. By separating the calculation of "blank" symbols from actual text tokens and skipping the expensive parts when a blank is likely, they achieve these gains with zero loss in recognition accuracy.
What Problem Does This Paper Address?
On-device Automatic Speech Recognition (ASR) is a balancing act between accuracy, latency, and battery life. The RNN-Transducer (RNN-T) has become the industry standard for this task because it's compact and handles streaming audio well.
While most research focuses on optimizing the "Encoder" (the part that processes audio) or the "Predictor" (the part that predicts the next word), this paper shines a light on the Joiner. The Joiner combines the outputs of the encoder and predictor to decide which character or word to output next.
Even though the Joiner is mathematically simple (often just a single layer), it is a massive efficiency bottleneck because it is executed 80x more frequently than the encoder during decoding. In mobile hardware, the constant "weight loading" (moving data from main memory to the processor) required for these frequent Joiner calls drains the battery rapidly.
Methods
The authors introduce Factorized Blank Thresholding, which relies on two key components:
- HAT-style Factorization: Borrowing from the Hybrid Autoregressive Transducer (HAT) architecture, they split the Joiner into two separate paths: one that calculates the probability of a "blank" symbol (indicating no new text at this millisecond) and one that calculates the probabilities of all actual vocabulary tokens (the "non-blank" path).
- Conditional Execution (The Threshold): RNN-T models are notoriously "spiky"—they output blanks for the vast majority of time steps. The authors propose calculating the blank probability first. If this probability is above a certain threshold (meaning the model is very sure there is no new text), the system skips the much more computationally expensive non-blank path entirely.
Crucially, because the "blank" Joiner is tiny (projecting to a single value), its weights can stay in the high-speed local cache (SRAM), while the much larger "non-blank" Joiner weights only need to be fetched from the power-hungry main memory (DDR) when absolutely necessary.
Datasets
The researchers evaluated their approach on two major benchmarks:
- LibriSpeech: A standard public dataset of 960 hours of read English speech.
- Large-Scale In-House Data: A massive dataset of 83 million hand-transcribed utterances (approx. 145,000 hours) collected from Meta Portal and Quest devices to test real-world robustness.
Key Insights and Ideas
- The Joiner is a Hidden Power Hog: While the Encoder has more parameters, the Joiner’s high invocation frequency makes it the dominant consumer of memory bandwidth.
- Exploiting "Spikiness": The authors found that for roughly two-thirds of Joiner calls, the model is so certain of a blank that the non-blank computation is redundant.
- Memory Bandwidth is the Real Enemy: On mobile chips, the energy cost of moving data (loading weights) is often much higher than the cost of the actual math. By reducing the number of times the large non-blank weight matrix is loaded, they achieve massive power savings.
- Accuracy is Preserved: Because the model is already very confident when it predicts a blank, skipping the alternative doesn't change the final transcript.
Key Results
The results were consistent across both public and private datasets:
- Speed: Achieved a 26-30% improvement in overall decoding speed (RTF).
- Power: Reduced on-device power consumption by 43-53%.
- Efficiency: Successfully skipped roughly 66% of non-blank Joiner invocations.
- Accuracy: There was zero Word Error Rate (WER) degradation on LibriSpeech and only negligible impact on out-of-domain in-house data when using a conservative threshold.
Limitations and Future Work
The main limitation identified is that a fixed threshold might not be optimal for all scenarios. For example, on "out-of-domain" data (audio that sounds very different from the training data), the model is less confident, and a fixed threshold might cause it to skip computations it actually needs, leading to slight accuracy drops.
The authors suggest two future directions:
- Dynamic Thresholding: Adjusting the threshold on the fly based on the model's confidence.
- Trainable Thresholds: Integrating the thresholding logic directly into the model's training process so it learns exactly when it can safely "skip" the hard work.
Summary updated