Abstract
Modern Automatic Speech Recognition (ASR) systems often struggle with "long-tail" entities such as technical jargon, medical terms, or specific names that are underrepresented in training data. TurboBias (GPU-PB) introduces a universal, GPU-accelerated framework for context-biasing that addresses this gap across all major model architectures, including CTC, RNN-T, and AED. By utilizing a novel phrase-boosting tree with a logarithmic weight distribution, TurboBias achieves significant accuracy improvements in both greedy and beam search decoding with minimal computational overhead, maintaining high performance even with context lists as large as 20,000 phrases.
Introduction
While general-purpose ASR models have reached impressive levels of accuracy, their performance on domain-specific vocabulary remains a challenge. Context-biasing techniques aim to mitigate this by providing the model with a list of relevant terms at inference time. Existing methods generally fall into two categories: Deep Fusion, which integrates context into the model architecture but requires expensive retraining, and Shallow Fusion, which applies a "boost" to specific words during decoding.
Shallow fusion is highly flexible but traditionally slow, often requiring computationally expensive beam search to be effective. TurboBias addresses these limitations by providing a production-ready, open-source framework that delivers high-accuracy context-biasing across all model types without the typical latency penalties associated with shallow fusion.
The TurboBias Framework
The core of TurboBias is the GPU-PB (GPU Phrase-Boosting) mechanism, which reimagines prefix tree traversal for speech recognition. The framework introduces several key innovations:
Logarithmic Depth Bonus
Traditional shallow fusion methods often reward the model only upon reaching the end of a phrase, which can lead to the model straying off-path during the intermediate steps of a long word. TurboBias introduces a "depth bonus" calculated as $c_0 * \beta + \ln(depth)$. By increasing the score incrementally at every step of a phrase, the model is significantly more likely to remain on the correct path, even during "greedy" decoding where only the single most likely token is selected at each step.
GPU-Optimized Architecture
To ensure scalability, the prefix tree is converted into a flat table structure optimized for GPU "index-select" operations. This allows the system to query the entire vocabulary for thousands of phrases simultaneously. The implementation leverages custom Triton kernels and CUDA graphs to minimize communication overhead between the CPU and GPU, ensuring that the biasing process adds negligible latency to the overall ASR pipeline.
Universal Model Support
TurboBias provides specific heuristics to adapt context-biasing for Connectionist Temporal Classification (CTC), Recurrent Neural Transducer (RNN-T), and Attention Encoder-Decoder (AED) models. For AED models, which are prone to "hallucinations" under strong bias, the framework includes dynamic boosting for the End-of-Sentence (EOS) token to ensure proper termination.
Experimental Evaluation
The researchers evaluated TurboBias across three distinct domains: Earnings21 (financial calls), MultiMed (medical data), and CSTalks (technology keynotes). The ASR models were pre-trained on a 24,000-hour corpus of open-source English speech.
The results demonstrate that TurboBias significantly improves the F-score—a metric combining precision and recall for keywords—by 8–10% in greedy mode and 17–23% in beam search. Notably, the framework makes greedy decoding a viable and highly effective strategy for context-biasing, which was previously thought to require beam search. Beyond keyword recognition, the overall Word Error Rate (WER) improved as the models became more adept at identifying difficult technical terms.
Scalability and Efficiency
A major hurdle for context-biasing systems is performance degradation as the number of keywords increases. TurboBias demonstrates remarkable robustness in this regard.
As shown in the scalability analysis, increasing the context list from 200 to 20,000 phrases resulted in only a 5% impact on the total system runtime (RTFx). This efficiency is achieved through the GPU-optimized data structures that allow for parallelized tree traversal, making TurboBias suitable for large-scale production environments where users may have extensive personal or domain-specific dictionaries.
Conclusion
TurboBias represents a significant advancement in ASR context-biasing, offering a universal solution that balances accuracy, speed, and scalability. By proving that greedy decoding can be highly effective with the right weight distribution, it opens the door for high-performance, low-latency biasing in streaming applications. The framework is now open-sourced as part of the NVIDIA NeMo toolkit, providing the research and development community with a powerful tool for improving ASR performance on long-tail entities.
Summary updated