Abstract
RNN-Transducer (RNN-T) models have become synonymous with streaming end-to-end ASR systems. While they perform competitively on a number of evaluation categories, rare words pose a serious challenge to RNN-T models. One main reason for the degradation in performance on rare words is that the language model (LM) internal to RNN-Ts can become overconfident and lead to hallucinated predictions that are acoustically inconsistent with the underlying speech. To address this issue, we propose a lightweight adaptive LM discounting technique AdaptLMD, that can be used with any RNN-T architecture without requiring any external resources or additional parameters. AdaptLMD uses a two-pronged approach: 1) Randomly mask the prediction network output to encourage the RNN-T to not be overly reliant on its outputs. 2) Dynamically choose when to discount the implicit LM (ILM) based on rarity of recently predicted tokens and divergence between ILM and implicit acoustic model (IAM) scores. Comparing AdaptLMD to a competitive RNN-T baseline, we obtain up to 4% and 14% relative reductions in overall WER and rare word PER, respectively, on a conversational, code-mixed Hindi-English ASR task.
Introduction
The RNN-Transducer (RNN-T) architecture is currently the industry standard for streaming, end-to-end Automatic Speech Recognition (ASR). By fusing an acoustic encoder (Transcription Network) and a language encoder (Prediction Network), the RNN-T provides a robust framework for real-time transcription. However, these models frequently struggle with "hallucinations," particularly when encountering rare words or out-of-domain vocabulary.
Because the internal language model (ILM) is trained jointly with the rest of the network on the same dataset, it often becomes overconfident. In scenarios involving rare names or technical terminology, the ILM may override the acoustic signal, "correcting" a rare word to a more frequent but acoustically similar one (e.g., misrecognizing "Canara Bank" as "Amazon"). This over-reliance on linguistic priors at the expense of acoustic evidence limits the utility of RNN-Ts in specialized domains.
Methodology: The AdaptLMD Approach
To mitigate ILM overconfidence, we introduce AdaptLMD, a lightweight technique that functions without external data or additional parameters. The method operates through two primary strategies: training-time regularization and inference-time adaptive discounting.
1. Training with LM Masking
To prevent the joint network from becoming overly dependent on the Prediction Network, we implement random masking during the training phase. By replacing the Prediction Network's output with a zero vector at a specific probability, the model is forced to learn how to generate accurate predictions using the acoustic signal alone. This encourages the joint network to treat the linguistic signal as a guide rather than an absolute constraint, making the system more robust to spurious linguistic outputs.
2. Adaptive Discounting during Decoding
During inference, AdaptLMD dynamically adjusts the influence of the ILM score. The discount factor is determined by two real-time metrics:
- Discrepancy: We calculate the KL-divergence between the standalone Implicit Acoustic Model (IAM) and the Implicit Language Model (ILM). A high divergence suggests the two components disagree, signaling a potential hallucination where the LM may be deviating from the audio.
- Rarity: The system maintains a rolling average of ILM probabilities ($P_{roll}$). If the recently predicted tokens exhibit low probability—indicating a rare or unfamiliar sequence—the system increases the discount on the ILM, placing greater trust in the acoustic signal.
Experimental Setup
We evaluated AdaptLMD on a proprietary code-mixed Hindi-English (Hi-En) dataset comprising 628 hours of call-center speech. The data spans four distinct domains: Banking, Insurance, Retail, and Telco. To test generalization, we conducted both in-domain evaluations and out-of-domain (leave-one-domain-out) experiments to observe how the model handles unseen vocabulary.
Results and Discussion
AdaptLMD demonstrates significant improvements over standard RNN-T baselines, particularly in the recognition of rare words:
- Rare Word Accuracy: On out-of-domain tasks, AdaptLMD achieved up to a 14% relative reduction in Phoneme Error Rate (PER) for rare words. This metric was chosen because rare words were often missed entirely by the baseline (resulting in 100% WER), making PER a more granular measure of partial success.
- Overall Performance: The method yielded a 4% relative reduction in overall Word Error Rate (WER).
- Comparison to Existing Methods: AdaptLMD consistently outperformed the "Density Ratio" method. Notably, while Density Ratio requires an external language model, AdaptLMD achieves superior results using only the internal components of the RNN-T.
- Ablation Studies: Our analysis confirmed that both training-time masking and decoding-time discounting are essential; neither technique alone provides the full performance gain.
Qualitative analysis shows that AdaptLMD successfully prevents common word substitutions (e.g., correcting "business" back to the intended rare word "discount" in specific contexts), effectively "tempering" the LM's influence when it is most likely to err.
Conclusion and Future Work
AdaptLMD provides a parameter-free, computationally efficient solution to the hallucination problem in RNN-Ts. By treating the internal acoustic and linguistic components as independent experts and monitoring their divergence, we can significantly improve the recognition of rare and out-of-domain words.
Future research will focus on automating the tuning of hyperparameters ($\lambda$ and $\rho$) which currently require a validation set. Additionally, we aim to explore the efficacy of AdaptLMD in morphologically rich languages and purely monolingual contexts to determine if these hallucination patterns are universal across different linguistic structures.
Summary updated