Hearing Where a Name Comes From: Origin-Aware Entity Recovery in Speech Translation

Speech translation systems get most words right and then fumble the names. A name like "Szczepański" or "Nguyen" enters the system as sound. The English ASR (automatic speech recognition) model, which converts speech to text, was built around English spelling rules, so it spells the name wrong. Every later stage inherits that mistake. Text translation research has long known the remedy for names: first guess where a name comes from (Polish, Vietnamese, French), then apply that language's sound-to-spelling rules. That guess is called name origin recognition. All of that work, however, assumes the name is already correctly written down. In speech, the written form is exactly what is missing.

We propose to infer a name's likely origin directly from the speech encoder's internal representation of the spoken name and its surrounding words. The result is a soft probability over origins, not a hard label. Nothing is decided early, so an error in the origin guess cannot cascade the way a transcription error does. We then use that probability in two places. It reweights the pronunciations used to match spoken names against a dictionary, and it steers a small spelling generator for names the dictionary lacks. The whole addition must run at a small fraction of the latency cost of existing text-side origin pipelines.

Core Claim

The claim has three parts, each testable:

  1. A distribution over origins, predicted from speech-encoder states over a detected name span and its context, recovers foreign-origin person names better than two alternatives. The first predicts origin from the ASR transcript, where a misspelled string gives a text classifier corrupted input. The second skips origin and enumerates every origin's pronunciation equally.
  2. Keeping the origin decision soft preserves the error tolerance that soft interfaces are known to buy.
  3. The mechanism adds a single-digit percentage to inference latency, because it runs only on the roughly one entity per utterance that a detector already isolates.

Where This Sits in Prior Work

What the origin literature establishes. It shows that one averaged set of sound-to-spelling rules fails on mixed-origin names:

  • Haizhou Li et al. documented phonetic distortion under uniform rules.
  • 47/2026-1 et al. showed that Japanese and Euramerican names have sharply different character distributions.
  • Jong-Hoon Oh et al. got their largest English-to-Korean gains by isolating Greek-origin terms.

Three lines supply our origin model:

  • Discriminative classification. Min Zhang et al. classified origin with a maximum-entropy model over character n-grams.
  • Using context for short names. Min Zhang et al. used surrounding document context to compensate for how little signal a short name carries. We borrow this directly, by pooling encoder states over a context window rather than over the name alone.
  • Origins learned without labels. Hagiwara et al. and Hagiwara et al. treated origin as a hidden variable learned with expectation-maximization (EM) under a Dirichlet prior. Tingting Li et al. and the tightly-coupled clustering and alignment model learned alignment and clustering jointly. These let us define origin clusters where nationality labels are missing.

All of this work operates on clean strings. We move the origin decision onto acoustic states.

The closest prior systems. Two papers are nearest, and each is missing one half of what we propose.

Extract and Attend is a text-only translation system. It predicts a name's nationality and generates candidate transliterations from nationality-specific models. It then lets the translation decoder attend softly to those candidates through a decoder prefix, rather than hard-replacing them into the output. The paper reports about 9% better Chinese-to-English name translation and 35% fewer entity errors, but average inference time rises from 389ms to 624ms per sentence. We reuse its soft-prefix idea. We differ in taking origin from speech, in keeping the origin decision probabilistic, and in replacing its multiple per-nationality models with one shared generator.

Named Entity Detection and Injection for Direct Speech Translation is a speech system built in two stages. A detector matches dictionary entries against shared speech and text encoder states; it retrieves about 1.2 candidates per utterance. An injection module then biases the decoder toward those candidates. On NEuRoparl-ST it cuts person-name errors by 31.3%, but it loses 1.3–2.0 BLEU and fails on pronunciations that deviate from standard letter-to-sound rules. Those deviations are exactly the foreign-origin case. It is the system we build on and must beat.

Motivation for keeping the interface soft. Work on cascaded errors motivates soft interfaces. Attention-Passing Models pass attention vectors instead of text between the recognition and translation stages. Under 10% transcript corruption, translation quality drops only 10%, against 26% for a cascade. We apply the same logic one level up: the origin decision itself is passed softly.

A constraint on the method. Soft Alignment of Modality Space shows that forcing speech states to match text states pointwise damages the text space, cutting text translation from 39.1 to 34.8 BLEU. A distribution-level adversarial alignment avoids this damage. We therefore do not force speech name representations onto text name embeddings. We also test whether adversarial alignment erases the origin information we depend on.

Parameter-cheap conditioning. Tuning LayerNorm in Attention shows that re-scaling normalization layers alone can adapt a model to a new input distribution cheaply. We use that idea to condition one shared spelling generator on origin.

Research Program

The base system. We start from the Detect-and-Inject architecture on a direct speech translation model.

The origin head. For each span the detector flags, we pool encoder states over the span plus a fixed context window of neighbouring frames. A small classifier maps this to a posterior $p(k \mid h_s)$ over $K$ origin clusters, where $h_s$ is the pooled representation of span $s$. The clusters come from two sources. Where person names can be linked to Wikidata, their nationality serves as a label. Otherwise we use Dirichlet-mixture clusters learned over the dictionary's names, following Hagiwara et al. and Li et al. The head is trained on names in the training data whose origin is known or inferred.

Origin-weighted matching. Each dictionary candidate $c$ is expanded into $K$ pronunciations $\pi_k(c)$, one per origin's letter-to-sound rules. The detector's score becomes

$$\text{score}(c,s)=\log \sum_{k=1}^{K} p(k \mid h_s)\; p\big(\text{match} \mid \pi_k(c), h_s\big),$$

so a pronunciation variant only counts in proportion to how plausible its origin is for this utterance. The intended effect is to recover names whose pronunciation standard English rules miss, without the false matches that equal-weight expansion would add.

Out-of-dictionary names. A single character-level spelling generator reads a phone sequence decoded from the span. It carries $K$ sets of LayerNorm gain and bias parameters inside its attention blocks, mixed by $p(k \mid h_s)$. Its top candidates join the injection module's bias list, which the decoder attends to softly. It never hard-replaces text.

Latency. The generator runs only on detected spans, which is where we expect the latency savings over Extract and Attend to come from.

A probing analysis. We train linear probes to predict origin from encoder states under three conditions: no alignment, hard contrastive alignment, and soft adversarial alignment. This answers the open question of whether aligning speech and text distributions erases origin-specific structure. It also tells us which encoder to build on.

Baselines

Direct model with no entity handling. This is the naive reference: the direct speech translation model alone. It fixes the floor for name accuracy and the ceiling for BLEU.

Detect-and-Inject. This is the strongest existing method on the exact question, since it is the only system in the file that recovers named entities from speech against a dictionary. It reports a 31.3% person-name error reduction and up to 7.1% entity-accuracy gain on NEuRoparl-ST, at a cost of 1.3–2.0 BLEU. We found no reported absolute person-name accuracy on Lacuna, so we re-run it ourselves. It gets the same encoder, training data, dictionary, and hyperparameter-search budget as our system. Beating it means two things together. First, a statistically significant gain in person-name accuracy on the non-English-origin subset. Second, a BLEU loss relative to the no-entity model smaller than its own.

Strong cascade. An in-domain fine-tuned recognition-plus-translation cascade following Does Joint Training Really Help Cascaded Speech Translation?, which reports 30.5 BLEU on MuST-C En-De tst-COMMON. We then add text-side origin classification and nationality-specific transliteration in the style of Extract and Attend, run on the 1-best transcript. This is the "text origin" competitor. It also serves as our latency reference, alongside Extract and Attend's reported 389→624ms.

Joint entity tagging. The parallel-head model from Joint Speech Translation and Named Entity Recognition, which reports a 0.4–1.0 F1 gain over a cascade at negligible cost. It shows what a zero-overhead extra prediction head buys when it predicts entity type rather than origin.

Ablations. These isolate the key ingredient:

  • equal-weight pronunciation expansion (the $K$ variants with a uniform posterior);
  • a hard argmax origin in place of the posterior;
  • an origin posterior computed from the ASR transcript instead of encoder states;
  • $K$ separate spelling generators in place of the mixed-LayerNorm generator.

All ablations share one compute and tuning budget.

Why not just…?

Why not just classify origin from the ASR transcript, as text systems do? A misspelled name is precisely the case where a text origin classifier sees garbage. SpeechGPT makes the general point that downstream text modules cannot see acoustic evidence. The attention-passing results quantify the cost: a 26% versus 10% drop under corrupted intermediate text. Still, a strong fine-tuned recognizer may spell many foreign names well enough. The transcript-origin ablation settles this directly. We will report accuracy separately on names the recognizer spelled wrong, which is where we expect the gap.

Why not just add every origin's pronunciation to the dictionary and let Detect-and-Inject match against all of them? This is the most serious threat, and it is cheap. Its expected weakness is precision. With $K$ variants per entry, spurious matches grow. Each spurious match becomes an injected bias, which is plausibly the source of Detect-and-Inject's BLEU loss. Expansion also does nothing for names absent from the dictionary. The uniform-posterior ablation tests this head-on. If it matches our system on in-dictionary names, we will say so. The contribution would then narrow to out-of-dictionary spelling and to false-match reduction measured as injected candidates per utterance.

Why not just train a bigger multilingual model with language tags and let it learn names implicitly? Tags help. The Indic transliteration study reports about 20% gains from multilingual training and 6–7% from source tags. Yet foreign-origin words stayed systematically harder even there. A language tag marks the utterance's language, not the origin of one name inside it. Direct models also degrade in low-data regimes, per Sperber et al. and CoVoST 2 (BLEU < 1 for low-resource bilingual pairs), and rare foreign names are a low-data regime. Our no-entity baseline trained multilingually measures what implicit learning achieves.

Why not just fine-tune the whole detector or add LoRA adapters per origin? That is possible, but it multiplies parameters and memory. The LayerNorm-tuning paper reports LoRA underperforming normalization tuning in vision-language adaptation, while using 41.9% more trainable parameters. That result is untested for speech and for spelling generation. The separate-generator ablation shows whether the cheap mixture loses accuracy.

Proposed Experiments and Decisive Checks

Data and labels.

  • NEuRoparl-ST (en→es, en→fr, en→it) is the main test bed. We link person entities to Wikidata nationality, which gives an origin-annotated test subset that splits names into English-origin and other-origin.
  • MuST-C En-Ru supplies a cross-script setting, where names must be transliterated into Cyrillic. We will manually annotate a person-name subset of tst-COMMON with correct Russian spellings and origins.
  • The Indic test set, split into 928 foreign-origin and 572 Indian-origin words, evaluates the spelling generator in isolation on clean input. This separates generator quality from speech effects.

Metrics. Person-name accuracy by origin subset, BLEU, injected candidates per utterance, and wall-clock latency per sentence on fixed hardware.

Decisive check 1: speech versus transcript. The audio-derived posterior must beat both the transcript-derived posterior and uniform expansion on the non-English-origin subset. The gap should be largest on names the recognizer misspelled. If it is not, the core claim fails.

Decisive check 2: soft versus hard. Under synthetic corruption of origin labels during head training, the soft posterior should degrade less than the hard argmax. This mirrors the attention-passing noise experiment.

Decisive check 3: cost. Added latency over Detect-and-Inject should stay under 10%, against Extract and Attend's 60.4%. The BLEU loss should also shrink relative to Detect-and-Inject's 1.3–2.0.

The probing study. This determines whether origin information survives in the encoder under hard and soft alignment. If soft alignment erases origin separability, that is a useful negative result for the alignment literature. It would also tell us to read the origin head from lower, unaligned layers.