Speech recognizers fail most often on rare words: product names, executive names, drug names, patient-specific terms. The standard fix is contextual biasing. Before decoding, the system receives a list of words likely to be spoken, taken from a document tied to the recording such as slides or a patient chart, and it boosts those words during decoding. The weak point is the list. On real earnings calls, the attached documents contained only 39% of the product names actually spoken, and a recognizer that used those documents perfectly would still leave a large share of rare-word errors in place. We propose to fill the missing part of the list from related recordings. The system builds a population graph, a network in which each node is one recording together with its documents (one call, or one patient encounter) and edges link similar ones. The biasing list for a new recording is then its own context plus the entities that its graph neighbors mentioned, and a small calibrated model decides which neighbor entities are worth including.
The two-sentence version is this. Each recording's own documents miss many of the rare words it will contain, and recordings that resemble it tend to have used those words. We therefore retrieve candidate words from similar past recordings and rank them by how likely each is to be spoken, and we test whether this closes the coverage gap without flooding the recognizer with false alarms. The idea joins three threads. The population-graph literature supplies the retrieval structure. Multimodal fusion supplies the combination of document text, speech, and relational context. Long-tail recognition failure supplies the target and the evaluation.
Core Claim
Contextual speech recognition has an under-studied gap in coverage, separate from the familiar gap in utilization. ConEC built an earnings-call benchmark whose real supplementary documents cover 82% of spoken person names but only 39% of product names. With a transducer model (a standard streaming recognizer) and shallow fusion, which adds a score bonus during beam search when a hypothesis matches a listed word, rare-word word error rate (WER) moves from 26.02% to 24.84%. An oracle that exploits the document context perfectly reaches 18.72%. That oracle is bounded by what the documents contain. Our claim is that entities from a recording's population-graph neighbors cover a substantial fraction of the missing rare words while adding a controllable number of distractors. It follows that neighbor-expanded lists should push rare-word error below what own-context biasing can reach, including the own-context oracle.
The claim draws on prior work in three roles. ConEC motivates it: it treats each call in isolation and flattens documents into a bag of words. Are Population Graphs Really as Powerful as Believed? constrains it. That paper shows that hand-built patient graphs used for classification rarely beat a tuned random forest, because the constructed edges are noisy and neighbors often have different labels. We take this seriously. We do not propose message passing to classify recordings. We use the graph only as a retrieval index, where the cost of a noisy edge can be measured directly as a false alarm. Retrieve and Copy and CB-Whisper show that very large lists dilute attention and cause the recognizer to insert words that were never said. This constrains how many neighbor entities we can add, which is why a scoring model sits between retrieval and biasing.
Research Program
The system has three parts, and each is deliberately simple.
The first part is the graph. Each node is a recording with its attached text. In ConEC, the text is the slide deck, press release, and attendee list, plus company and sector metadata. Edges come from k-nearest neighbors over sentence embeddings of that text, with shared-entity Jaccard overlap as the alternative edge type. This mirrors the entity-overlap patient graph of PM²F²N, but here the graph feeds a recognizer rather than an outcome classifier. Neighbors must precede the target recording in time, and their entities come from their reference transcripts. This is the realistic deployment setting, where last quarter's calls or past encounters have already been transcribed and reviewed.
The second part is an entity scorer. For every candidate entity in the union of own context and neighbor transcripts, a gradient-boosted classifier predicts the probability that the entity is spoken in the target recording. Its features are own-context membership, the similarity-weighted count of neighbors that spoke the entity, global corpus frequency, entity type, and the phonetic distance to the nearest other candidate. We include that last feature because Contextual Speech Recognition with Difficult Negative Training Examples shows that sound-alike distractors are what break biasing. The scorer is trained on earlier recordings. The biasing list is the top-$K$ candidates by predicted probability, and each candidate's shallow-fusion bonus scales with that probability.
The third part is the recognizer. We keep ConEC's icefall Zipformer transducer with shallow fusion unchanged, so that any gain comes from the list alone. We then repeat the experiment with Whisper Large using prompt-based biasing. This separates the question of which words go on the list from the question of how the decoder consumes them.
A secondary clinical setting tests whether the idea transfers. The file has no public corpus pairing clinical audio with patient records. We therefore build a read-speech or synthesized-speech test set from MIMIC-III discharge summaries. Each patient's graph uses structured data recorded before the note, namely diagnosis codes, medications, and vitals. We label this set openly as a proxy. Because WER ignores clinical severity, with only 10.5% agreement with clinician preferences against 75.9% for Clinical BERTScore, clinical results are also reported with that metric.
Baselines
The naive reference points are recognition with no biasing and biasing with the recording's own documents. On ConEC these give rare-word WER of 26.02% and 24.84% respectively for the Zipformer with shallow fusion, as reported by ConEC. We rerun both in our environment and confirm the reproduction before reporting any comparison. The reported synthetic-context result of 23.43% and the own-context oracle of 18.72% serve as reference ceilings. Beating 24.84% at equal list size is the minimum bar. Getting below 18.72% is the result that would show coverage, not utilization, is the binding constraint.
The strongest competitor on list construction is acoustic-first retrieval over a large catalog in the style of Retrieve and Copy. It uses the audio itself to query approximate nearest-neighbor search over every entity in the corpus, and it bypasses the need for any relational prior. We found no reported result on Lacuna for Retrieve and Copy on ConEC, so we reimplement it with the same entity catalog, the same recognizer, and the same final list size $K$. Both methods receive the same number of hyperparameter trials on a shared validation split. We also run a global-frequency list, which takes the most common rare entities in the corpus or sector without per-recording similarity, and an all-neighbors union that has no scorer.
Two ablations isolate the key ingredients. The first replaces graph neighbors with random same-sector recordings, which shows whether similarity matters. The second removes the scorer and admits neighbor entities by raw count, which shows whether calibration controls false alarms. A third variant swaps the k-nearest-neighbor retrieval for a learned graph neural network that re-weights edges. It stays in the study only if it beats plain k-nearest neighbors, a direct test of the negative result from Are Population Graphs Really as Powerful as Believed? in a retrieval setting.
Why not just…?
Why not just make the own-context list bigger by including every entity from the whole corpus? Coverage would rise, but both Retrieve and Copy and CB-Whisper report that large lists cause attention dilution and hallucinated hotwords. We will plot rare-word WER and false-insertion rate against list size for both the full-catalog list and the neighbor-scored list. The proposal only earns its place if its curve dominates at every $K$.
Why not just retrieve candidates from the audio itself, with no graph? This is the real threat. Acoustic retrieval sees exactly what was said, while the graph only sees what similar recordings said. Acoustic retrieval is weakest, however, on sound-alike distractors and on garbled rare words, which is exactly where a prior from similar recordings adds independent evidence. We run both methods and their combination, in which graph-scored candidates are re-ranked by acoustic match. If acoustic retrieval alone matches the combination, we will report that the graph is unnecessary.
Why not just use a global or sector-level frequency list? Sector membership is a crude population graph. If it matches similarity-based neighbors, the per-recording graph adds nothing, and the random-same-sector and global-frequency baselines test exactly this. For the clinical setting, KnowRare gives indirect evidence against pooling: it found that aggregating all ICU conditions hurt rare-condition prediction while selecting the top-$k$ similar conditions helped. That evidence comes from outcome prediction, not speech recognition, so it motivates the test rather than settling it.
Why not just scale the recognizer? Scale does not close the gap: ConEC reports that Whisper Large still errs on 28.9% of person names and 19.6% of organization names. Our Whisper experiments test whether neighbor-filled lists help a large model as much as a small one.
Proposed Experiments
The first analysis requires no recognizer at all and should be run before anything else. It measures, on ConEC, the fraction of spoken rare entities that each list-construction method covers at list sizes $K$ from 50 to 2000, broken down by entity type. If neighbor lists do not raise product-name coverage well above 39% at modest $K$, the project stops there. If they do, we run the full recognition comparison. We report rare-word WER, entity recall, and false-insertion rate on a time-ordered split, with bootstrap confidence intervals over calls.
The decisive checks are three. The first is whether neighbor-scored lists reach rare-word WER below the 18.72% own-context oracle. The second is whether they beat acoustic-first retrieval at matched $K$. The third is whether random same-sector neighbors perform clearly worse than similarity neighbors. The MIMIC-derived proxy then tests whether the same ordering holds for clinical terms, scored with both WER and Clinical BERTScore, with the caveat that read or synthesized speech overstates acoustic clarity.
References
- ConEC: Earnings Call Dataset with Real-world Contexts for Benchmarking Contextual Speech Recognition
- Are Population Graphs Really as Powerful as Believed?
- Retrieve and Copy: Scaling ASR Personalization to Large Catalogs
- CB-Whisper: Contextual Biasing Whisper Using Open-Vocabulary Keyword-Spotting
- P\textM²\textF²N: Patient Multi-view Multi-modal Feature Fusion Networks for Clinical Outcome Prediction
- Contextual Speech Recognition with Difficult Negative Training Examples
- Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings
- Bridging data gaps of rare conditions in ICU: a multi-disease adaptation approach for clinical prediction