Automatic Speech Recognition (ASR) models have achieved remarkable accuracy on everyday language, but they frequently stumble when encountering rare words, such as specialized jargon, unique product names, or contact lists. To solve this, researchers use contextual biasing—providing the ASR system with a dynamic "cheat sheet" of expected terms (or "hotwords") during decoding. This context can be injected via shallow fusion, deep cross-attention layers, or specialized adapters.
However, giving a model a list of words it might hear introduces a fundamental trade-off. When an ASR system is aggressively nudged to look for specific terms, it often becomes over-eager. It begins to hallucinate these rare words in place of common, everyday speech, a phenomenon known as contextual over-biasing or false-positive biasing.
Understanding why over-biasing happens and how to mitigate it is essential for building personalized speech interfaces that remain robust during general conversation.
The Tug-of-War Between Acoustics and Context
To understand why models over-bias, think of ASR decoding as a tug-of-war between two forces: the acoustic evidence (the actual sounds extracted from the audio) and the contextual prior (the text phrases we want the model to favor).
When we apply contextual biasing, we artificially boost the probabilities of the phrases in our context list. If these biasing weights are too aggressive, or if the model's internal distributions become excessively sharp, the contextual prior completely overpowers the acoustic evidence. As demonstrated by Yinglu Li et al. in CB-Whisper: Contextual Biasing Whisper Using Open-Vocabulary Keyword-Spotting, the model begins to over-prioritize the text prefix over the actual acoustic signal.
Without a mechanism to penalize or revert incorrect matches, the system forces a pre-defined term into the transcript even when the speaker's actual utterance only partially matches the keyword prefix, as noted by Namkyu Jung in Spell my name: keyword boosted speech recognition.
This imbalance manifests in three primary ways:
1. Phonetic Distractors and Homophones
The risk of over-biasing skyrockets when a common word sounds nearly identical to a rare word on the bias list. In Mandarin speech recognition, for example, Chengxu Yang et al. show in Contextual Biasing with Confidence-based Homophone Detector for Mandarin End-to-End Speech Recognition that aggressive biasing causes the system to mistakenly replace common words with phonetically similar rare terms. As the size of the biasing list grows, this "volume sensitivity" introduces more acoustic distractors, making it incredibly difficult for the model to distinguish between a valid hotword and a common homophone (Yue Gu et al. in Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling).
2. The Anti-Context Blindspot
Why can't the model simply learn when to ignore the list? The answer lies in how these systems are trained. Standard training routines often present the model with pairs of audio and text where the target hotword is always present in the provided context list. This creates a massive blindspot.
As argued by Xiaoqiang Wang 0006 et al. in Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation, models frequently fail in "anti-context" scenarios—where the spoken word is entirely absent from the list—because they have never been explicitly trained to reject the bias list and trust the baseline acoustics.
3. Distributional Overconfidence
When biasing lists are small or highly specific, the resulting probability distributions can become excessively sharp (Chao Zhang et al. in Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition). The model becomes overconfident in its narrow search space, leading to insertion errors where it hallucinates words from the list during periods of silence or ambiguous background noise.
The Cost of Over-Correction
When a system suffers from over-biasing, it experiences a severe drop in precision. Giovanni Motta et al. observed in Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities that while aggressive boosting dramatically improves the recall of rare names, it triggers frequent false alarms, scattering personalized terms across general speech contexts where they do not belong.
This over-correction causes a noticeable regression in the Word Error Rate (WER) for non-contextual speech. This is true regardless of the modality providing the context:
- Visual Context: In robotics, using a camera to bias an ASR system toward objects in the immediate environment can backfire if the user talks about something outside the room (Pradip Pramanick et al. in Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?).
- Metadata Context: In open-domain video transcription, biasing models toward video titles or descriptions risks hallucinating those terms when they are never spoken aloud (Mahaveer Jain et al. in Contextual RNN-T for Open Domain ASR).
Architectural Remedies: Restoring the Balance
To prevent models from being hijacked by external text lists, researchers have developed several architectural mechanisms designed to protect the integrity of the acoustic signal.
[Audio Input] ---> [Entity Detector / Slot Classifier]
|
+---> (Context Found?) ---> [Activate Biasing] ---> [Output]
|
+---> (No Context) ---> [Bypass Biasing] ---> [Output]
Dynamic Gating and Slot-Triggering
Instead of leaving the contextual biasing module active at all times, systems can use an "intelligent toggle" that only activates biasing when a hotword is likely to occur.
Zhanheng Yang et al. implemented this in Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognition by introducing an Entity Detector module. This module acts as a gatekeeper, dynamically turning on the bias list only when the acoustic frames indicate an entity is being spoken.
Similarly, the authors of Model-Internal Slot-triggered Biasing for Domain Expansion in Neural Transducer ASR Models integrated a lightweight streaming classifier directly into a neural transducer. By triggering biasing only within specific "slots" (like a playlist name or contact slot), the system scales to massive catalogs without suffering from false-biasing on standard vocabulary.
Multi-Stage and Intermediate Grounding
Over-biasing often happens because a massive biasing score is applied as a blunt force at the very end of the decoding process (such as during beam search pruning). A more elegant solution is to distribute this alignment throughout the network.
In Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss, Muhammad Shakeel et al. introduced an Intermediate Biasing (IB) loss. By forcing the neural network to align bias phrases at multiple internal processing stages rather than just at the output layer, the model learns a gradual, representation-level integration of context, preventing the sudden distribution shifts that cause hallucinations.
Two-Pass Optimization and De-biasing
Another powerful strategy splits the responsibility of transcription into two distinct phases: one optimized for recall, and the second optimized for precision.
Aditya Gourav et al. pioneered this approach in Personalization Strategies for End-to-End Speech Recognition Systems. Their system uses a highly sensitive "subword lookahead" biasing technique in the first pass to ensure rare terms are not missed. Then, a secondary "de-biasing" optimization phase runs over the hypotheses, acting as a sober editor that strips away false positives and restores common words if the acoustic evidence for the hotword is weak.
Key Foundational Work
Shallow-Fusion End-to-End Contextual Biasing (2019) — contributed 1 concept elements to this direction.
Personalization of End-to-End Speech Recognition on Mobile Devices for Named Entities (2019) — contributed 1 concept elements to this direction.
Automatic assessment of children's reading level (2007) — contributed 1 concept elements to this direction.