When someone tells a voice assistant "call Siobhan Nguyen," the name is almost the whole message. Usually there is no surrounding sentence to help a downstream model guess what the speech recognizer got wrong. Speech recognizers (ASR) mangle names they have rarely seen, often turning one name into several ordinary English words. Systems that try to repair this compare how the misheard text sounds against how each name in the user's contact list or catalog sounds. That comparison is only as good as the catalog's pronunciations. Those pronunciations come from grapheme-to-phoneme (G2P) conversion, which predicts a pronunciation from spelling. G2P is known to be weakest exactly on proper names, because names keep spellings from their language of origin. German and Italian "sch" are pronounced differently, and the same surname may be said natively by one family and anglicized by another.
This note proposes to fix the matching at the catalog side rather than the transcript side. The misheard transcript consists of ordinary English words, whose pronunciations a generic G2P predicts reliably. The hard part is knowing every plausible way each catalog name might actually be spoken. We will give each catalog name a small set of pronunciation variants, weighted by a predicted language of origin and covering both native and anglicized forms. A lightweight tagger, a BiLSTM-CRF, will then find the span of the transcript that is the mangled name and score it against those variants. The claim is that origin-aware pronunciation variants recover substantially more misheard non-Anglo names than generic phonetic keys or generic G2P alternatives, at the same false-correction rate and within a few milliseconds. The gain should come specifically from names whose spelling misleads English letter-to-sound rules.
Why this gap is real
The failure is severe and well documented. Why Aren't We NER Yet? analyzes a large text-based entity tagger (RoBERTa-Large) on speech transcripts. It shows that when the recognizer substitutes even one token inside an entity, the tagger finds the span only 18% of the time. On SLURP single-turn home-assistant commands, SLURP reports entity SLU-F1 of 70.84 for its best pipeline versus 78.19 on perfect transcripts. In these short commands the context-based remedies do not apply, either because they need surrounding text or because they are slow. Piggyback borrows context from web search and costs 0.2 to 0.5 seconds per token. ERNIE and Falcon inject or query knowledge graphs, but only through surface strings that a misheard name no longer matches. These papers motivate the work: with sparse context, recovery has to go through sound.
Three systems already match by sound, and they define what this proposal must beat.
Entity Linking for Spoken Language links broadcast-news person mentions using phone, Metaphone and lattice features, a Metaphone being a coarse rule-based sound code. It raises accuracy from 0.55 to 0.78 at 51% word error rate. However, it uses an off-the-shelf G2P as a black box and works on context-rich news.
Hierarchical Multi-Stage Word-to-Grapheme Named Entity Corrector is the closest competitor. It corrects call and music requests against a catalog using word edit distance, then Double Metaphone codes, then a weighted grapheme distance, with a check against the recognizer's beam to reject false overrides. It reports 63% relative WER reduction on call requests and 14% on music, in 2–3 ms. Double Metaphone is an English-centric rule system, so it has no notion of origin.
Generative Annotation for ASR Named Entity Correction retrieves candidates acoustically from a datastore of synthesized speech. It then labels spans with a Whisper-based decoder, raising entity recall on loanwords from 61.82% (phonetic edit distance) to 87.47%, but with heavy generative decoding.
None of these three asks whether the catalog's pronunciations are right.
The method comes from two G2P papers. G2P Conversion of Proper Names Using Word Origin Information trains a character n-gram classifier for surname origin and smooths origin-specific G2P with a generic model, gaining 2.2% accuracy on 1990 US Census surnames matched to the CMU dictionary. How do you pronounce your name? re-ranks G2P hypotheses using transliterations in phonetically transparent scripts. Both papers evaluate pronunciation accuracy in isolation, never downstream recovery from speech. The research file also constrains the design in two ways. The origin classifier can be wrong, and the same name can be pronounced natively or anglicized. Committing to a single best pronunciation is therefore the wrong design. The catalog should hold a weighted set of variants.
Core Claim
Consider a catalog entry $e$ with origin posterior $p(o \mid e)$ from a character-level origin classifier. For each origin, an origin-conditioned G2P produces pronunciation hypotheses $v$ with probabilities $p(v \mid e, o)$. A generic English G2P always contributes an "anglicized" variant. A candidate transcript span $x$ is converted to phonemes $\phi(x)$ with the generic English G2P, which is reliable here because $x$ consists of English words. The match score is
$$ s(x, e) = \max_{v \in V_K(e)} \; \log\!\Big(\sum_{o} p(o \mid e)\, p(v \mid e, o)\Big) + \lambda \, \mathrm{sim}\big(\phi(x), v\big), $$
In this score, $V_K(e)$ is the top-$K$ variants by marginal probability, $\mathrm{sim}$ is a phoneme alignment score with learned substitution costs (cheap for confusable pairs such as /s/–/ʃ/), and $\lambda$ trades prior plausibility against acoustic closeness.
We predict that this score separates true from false matches better than Double Metaphone or generic G2P n-best. We further predict that the advantage concentrates in origin strata where English letter-to-sound rules mislead, such as Slavic, Vietnamese, Irish-Gaelic and Italian surnames.
Research Program
The work builds three components.
The first is the variant catalog builder. It combines the origin classifier and smoothed origin-specific G2P following Waxmonsky-style origin conditioning, trained on Census surnames with CMU pronunciations. It is extended so that it emits a weighted variant set instead of one string. Where transliterations are available, they re-rank the variants as in the transliteration paper. This runs offline when a contact list or catalog changes, so its cost is outside the latency budget.
The second is the span tagger. Recognizers routinely split one name into several words. A character-level BiLSTM-CRF tags the transcript with entity spans, where the BiLSTM encodes each word with left and right context and the CRF enforces valid begin/inside/outside tag sequences. Its emission features include, for each token, the best $s(x, e)$ over all spans of up to four tokens covering it. They also include the recognizer's per-word confidence. Confidence is used to trigger catalog search, not to suppress the entity. This deliberately inverts the design of Sudoh et al., who raised precision from 70.67% to 78.13% by down-weighting uncertain tokens, a strategy that discards the only content in a one-name command.
The third is a rejection step. It accepts a correction only when the best catalog score beats the transcript's own span by a margin tuned on development data. It also borrows the beam-agreement check of the hierarchical corrector to avoid overwriting correct names.
The tagger is trained on real recognizer output where available. Training is augmented with pseudo-errors made by passing names through the variant G2P and then a phoneme-to-grapheme decoder into English words. This follows the noise-generation recipe of Ohsugi et al., applied for the first time to span tagging. The pseudo-errors are produced from the origin-aware variants, so the training noise reflects how foreign names are actually misheard.
Baselines
The naive reference point is to change nothing. The recognizer's top transcript goes into the same BiLSTM-CRF entity tagger with no catalog matching, and an entity counts as correct only if the recognizer spelled it right. This measures how much recovery is possible at all. On SLURP, the reported pipeline number for this regime is SLU-F1 70.84 (SLURP). We will reproduce our own pipeline's number rather than assume it.
The strongest lightweight competitor is the hierarchical multi-stage corrector. It targets the same regime (short commands, fixed catalogs, millisecond budgets), and its phonetic stage is exactly the component we replace. Its reported result is 63% relative WER reduction on call requests and 14% on music. Those test sets are not public, so we will reimplement it from the paper's description. It will run on the same recognizer output, the same catalogs and the same development set, with its edit-distance weights and rejection threshold tuned under the same search budget as ours. Beating it means higher name recall at an equal false-override rate on the full test set, with no regression on Anglo-origin names, at comparable CPU latency (under 10 ms per utterance).
The strongest heavy competitor is acoustic retrieval plus generative labeling from Generative Annotation for ASR Named Entity Correction. It reports 87.47% entity recall on its BuzzWord set against 61.82% for phonetic edit distance. We will run it on our catalogs, with a speech-synthesis datastore built from the same names. We treat it as an accuracy ceiling to approach, not necessarily to beat, and we will report the latency of both systems. Matching most of its recall at a small fraction of its runtime would be a positive result.
Three ablations isolate the key ingredient. The first replaces the variant set with the single best generic English pronunciation, which removes both origin and variants. The second replaces it with the top-$K$ hypotheses of a generic G2P at the same $K$, which keeps variants but drops origin. This is the critical control, because extra variants alone raise recall. The third uses origin-conditioned single-best pronunciation, which keeps origin but drops variants and tests whether hedging between native and anglicized forms matters. A further ablation removes the CRF and scores fixed one-word spans, to test whether span tagging matters when names are split.
Why not just…?
Why not just swap Double Metaphone for a good neural G2P inside the existing hierarchical corrector? This is the most serious threat, and the generic-G2P ablations exist to test it. If generic top-$K$ G2P inside the reimplemented corrector matches our recall at equal false-override rates, the origin component adds nothing and we will say so. Our expectation rests on origin-conditioning results. There, generic models systematically mis-map origin-specific letter patterns. A generic G2P's alternatives are perturbations of an English reading, which rarely include the native reading. We will test this by stratifying recall by origin.
Why not just add more pronunciation variants until recall saturates? More variants raise recall but also raise false matches against other catalog entries, particularly in large catalogs such as the 500k-entry music case described in the hierarchical corrector. We therefore report recall at a fixed false-override rate and sweep $K$. The claim is that origin weighting puts better variants in the top few slots, so it should dominate at every matched $K$, not only at large $K$.
Why not just bias the recognizer toward the contact list or use a large end-to-end model? Biasing the recognizer toward the catalog is a legitimate complementary approach. The research file contains no reported numbers for it, so we will include a biased recognizer as an extra condition where the toolkit allows. Biasing uses the same catalog pronunciations, so origin-aware variants should help it too, and we will test that. Large generative correctors already exist (Generative Annotation) and are our heavy baseline. Their cost is the issue for on-device use, and SLURP reports that end-to-end models struggle on its lexically diverse commands.
Why not just train the tagger on pseudo-errors and skip catalog matching? The pseudo-error recipe of Ohsugi et al. improves robustness in general. However, a tagger without a catalog can at best find the mangled span; it cannot restore the spelling. Our pseudo-error-only condition will measure span detection and name recovery separately to show this split.
Proposed Experiments
Intrinsic pronunciation quality is measured on the 1990 Census surname plus CMU dictionary setup of the origin paper. The metric is variant-set coverage: how often the reference pronunciation appears among the top $K$ variants, broken down by origin class.
The main downstream evaluation is a new contact-calling test set. It contains a few thousand real recorded commands from multiple speakers, built around surnames sampled stratified by origin from the Census list. Each command is paired with 200-name and 20k-name contact catalogs, mirroring the catalog sizes of the hierarchical corrector. We record real audio rather than synthesize it, so that speech-synthesis pronunciation errors do not leak into the test. We also evaluate on the person and place entities of SLURP using SLU-F1 against its reported 70.84 pipeline and 78.19 clean-text figures. As a context-rich contrast, we evaluate on Earnings-21, where Szymanski et al. report micro F1 of 0.37. We expect the advantage to shrink there because context does more of the work.
The metrics are name recall at fixed false-override rates (1% and 5%), entity span F1, the ATENE-style entity-oriented view argued for in How to evaluate ASR output for named entity recognition? in place of raw WER, and CPU latency per utterance.
Three checks are decisive. First, the gain over generic top-$K$ G2P at matched $K$ must be significant overall and concentrated in non-Anglo strata. A uniform gain would suggest the improvement comes from something other than origin. Second, the gain must survive when the origin classifier is deliberately degraded, for example by evaluating on held-out origin classes. This shows that hedging across variants absorbs classifier errors, the failure mode the file flags for origin-conditioned pipelines. Third, the CRF span tagger must beat fixed single-word matching specifically on utterances where the recognizer split the name into multiple words. If the first check fails, the proposal reduces to "use a better G2P in the existing corrector," and we will report that result.
References
- Why Aren't We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts
- SLURP: A Spoken Language Understanding Resource Package
- Piggyback: Using Search Engines for Robust Cross-Domain Named Entity Recognition
- ERNIE: Enhanced Language Representation with Informative Entities
- Old is Gold: Linguistic Driven Approach for Entity and Relation Linking of Short Text
- Entity Linking for Spoken Language
- Hierarchical Multi-Stage Word-to-Grapheme Named Entity Corrector for Automatic Speech Recognition
- Generative Annotation for ASR Named Entity Correction
- G2P Conversion of Proper Names Using Word Origin Information
- How do you pronounce your name? Improving G2P with transliterations
- Incorporating Speech Recognition Confidence into Discriminative Named Entity Recognition of Speech Data
- Japanese ASR-Robust Pre-trained Language Model with Pseudo-Error Sentences Generated by Grapheme-Phoneme Conversion
- How to evaluate ASR output for named entity recognition?