Transferring Medical Speech Recognition Across Scripts by Relabeling Only the Shared Terms

A speech recognizer fine-tuned on medical dictation in one language does not help a hospital working in another language. Collecting medical audio in that second language is often blocked by privacy law, so it would help a great deal to reuse the first language's medical recordings. Much of what makes medical speech hard is shared across languages: drug names, Latin- and Greek-derived disease names, and English acronyms sound roughly alike in Polish, Hindi and Vietnamese. The difference is how each language writes them. Hindi writes "paracetamol" in Devanagari, or leaves it in Latin letters inside Devanagari text. Vietnamese often respells borrowed terms phonetically with its own diacritics. Our idea is to keep the source-language medical audio and rewrite only the domain terms in its transcripts into their target-language spellings. Everything else in the source sentence is left out of the target-language training signal. The model then learns from the source audio the one thing that actually carries over, which is how a shared term sounds, paired with how the target language writes it.

Put in two sentences: when you move medical speech recognition between distant languages and scripts, the transferable knowledge is concentrated in internationally shared terms, and the main obstacle is a spelling problem between character sets, not an acoustic one. Training a multilingual recognizer to emit target-script spellings for those terms only, using source-language medical audio and a text-only bilingual term list, should recover much of the medical-term accuracy that would otherwise need target-language medical audio.

Why the existing approaches stop short

Two papers already show that rewriting source transcripts into the target script helps cross-lingual speech recognition. Transliterated Zero-Shot Domain Adaptation for Automatic Speech Recognition pre-trains a wav2vec 2.0 model, with teacher–student pseudo-labels, to emit target-script transliterations of whole source utterances. It reports a 9.2% relative WER reduction on Mandarin→Cantonese and Russian→Czech. The authors note that the method weakens for distant languages, which is expected: a whole-sentence transliteration of a Russian sentence into Czech spelling is roughly meaningful, but the same operation from Polish into Devanagari produces strings no Hindi speaker would write. Exploring Transliteration-Based Zero-Shot Transfer for Amharic ASR transliterates Arabic and isiXhosa transcripts into Ge'ez script. It reaches 73.54% token error on Babel and finds that overlap in character distributions predicts transfer better than phoneme overlap. These two papers motivate our proposal, since they show output-script mismatch is a real barrier. They also supply our main prior-work baseline. Neither one touches a specialized domain, and both relabel everything. We relabel only the terms, and we target the regime they report as failing: distant language pairs.

A second line of work tells us where the failure is not. Unknown Script: Impact of Script on Cross-Lingual Transfer shows that, for text models, a byte-level BPE tokenizer (RoBERTa) handles an unseen script almost as well as its romanization: NER F1 is 0.57 on Fidel versus 0.56 on Latin. Romanization only rescues brittle subword vocabularies. Measuring Cross-lingual Transfer in Bytes finds that byte-level transfer is nearly insensitive to language distance (Spearman ρ = −0.147 with syntactic distance). Whisper already decodes byte-level BPE, so it can represent Devanagari and Vietnamese diacritics. These results constrain our design. If medical terms fail in transfer, the cause is not that the decoder cannot write the characters. The cause is that nothing has taught it which target spelling goes with a sound it learned from source-language audio. That is the specific gap our method addresses.

The domain papers set the stakes. Zero-shot Whisper-medium gets 24.03% WER on Polish clinical dictation, and fine-tuning on 83+ hours of medical speech brings it to 9.5% (A Comprehensive Polish Medical Speech Dataset for Enhancing Automatic Medical Dictation). On Vietnamese medical speech, XLSR-53 goes from 51.8% to 29.6% WER, but only with in-domain Vietnamese audio (VietMed). The best Hinglish clinical system adapted Qwen3-ASR on about 1,800 hours of Hindi speech and needed explicit Devanagari normalization to score fairly (Doctor or Patient?). Every one of these systems depends on target-language domain audio, and we want to remove that dependency. Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration shows that Whisper's output script can be steered at inference with a fixed activation vector. Its role here is narrow: it is evidence that script identity is cleanly separable in the decoder, which makes it plausible that spelling can be retaught for a small set of terms without disturbing the rest.

Core Claim

Our first claim is that, under zero-shot domain transfer between distant languages, term-selective relabeling will beat both naive cross-lingual fine-tuning and whole-transcript transliteration on medical-term recognition, without hurting general target-language accuracy. Our second claim is that the gain will appear almost entirely on terms that are pronounced similarly across languages ("loan terms") and not on terms the target language translates with native words. The second claim is the mechanistic test. If gains are spread evenly, the method is acting as generic text exposure rather than as an acoustic-to-spelling bridge, and our explanation is wrong.

Research Program

The setting is the zero-shot domain adaptation setup from the transliterated-ZSDA paper. The available data is source-language medical audio with transcripts (ADMEDVOICE, Polish), general target-language speech (FLEURS and Common Voice Hindi and Vietnamese, plus Viet-U for Vietnamese), and target-language medical text. No target-language medical audio is used.

The first component is a term lexicon built only from text. We extract medical terms from ADMEDVOICE transcripts by matching against ICD-10 and drug-name lists and against the dataset's six clinical categories. For each term we gather target spellings from bilingual medical glossaries and from LLM translation, with a native clinician verifying a sample. Each term is tagged as a loan term (similar pronunciation across languages) or a translated term (native target word). For Hindi, we keep every spelling attested in practice, Devanagari and Latin. Scoring uses Unicode NFC normalization, which What or Who is Multilingual Watson? showed matters for retrieval, along with the Devanagari normalization from the DISPLACE-M winner. This prevents us from counting script-formatting noise as error.

The second component is the training objective for a Whisper-medium backbone. We use Whisper-medium so that the Polish numbers above give a direct reference point. Each source utterance appears twice. Under the Polish language token it has its full native transcript. Under the target language token it has a decoder input sequence in which lexicon terms are rewritten in target spelling, and the loss is computed only on the term spans. Target general speech is trained normally. The total loss is

$$ \mathcal{L} = \mathcal{L}_{\text{src}} + \mathcal{L}_{\text{tgt-gen}} + \lambda\,\mathcal{L}_{\text{term}}, $$

where $\mathcal{L}_{\text{src}}$ is cross-entropy on full Polish medical transcripts, $\mathcal{L}_{\text{tgt-gen}}$ is cross-entropy on general Hindi or Vietnamese speech, $\mathcal{L}_{\text{term}}$ is cross-entropy restricted to target-spelled term tokens on source audio, and $\lambda$ is tuned on a held-out set of general target speech mixed with synthetic term prompts, never on target medical audio. The Polish loss keeps the encoder tuned to clinical acoustics, including the noisy treatment-room recordings in ADMEDVOICE. The masked term loss teaches the target-language mode a spelling for each sound without asking it to imitate Polish sentence structure.

The third component is analysis. We train linear probes for term identity on encoder and decoder layers to check whether loan terms from Polish and Vietnamese audio land near each other before and after training. We also measure general target WER to detect forgetting, which the domain papers flag as the main risk of fine-tuning.

Baselines

The do-nothing reference is zero-shot Whisper-medium decoding the target medical test sets. Next to it we report a target-general-only fine-tune, which adapts to Hindi or Vietnamese speech with no medical data, so that we can separate language adaptation from domain transfer.

The naive cross-lingual baseline fine-tunes on Polish medical audio with native Polish labels together with target general speech. It uses the same data as our method and lacks only the term relabeling. This is the comparison a skeptic would demand first, because multilingual encoders might transfer domain acoustics with no help.

The strongest existing method is whole-transcript transliteration in the style of Transliterated Zero-Shot Domain Adaptation and the Amharic transliteration study. Polish transcripts are fully transliterated into Devanagari or Vietnamese orthography, and the model is trained on them under the target language token. We run it on Whisper-medium and on an XLS-R CTC character model, which is closer to the original architectures. Its reported figure is a 9.2% relative WER reduction over wav2vec 2.0 on closely related pairs. No reported result was found on Lacuna for this method, or for any method, on zero-shot cross-lingual medical transfer to Vietnamese or Hindi. We therefore rerun it rather than quote it.

Two cheap competitors that use the same lexicon are also included. Lexicon prompting feeds the relevant target-spelled terms to Whisper as a decoding prompt. TTS synthesis renders target-language medical sentences containing lexicon terms with an off-the-shelf Hindi or Vietnamese synthesizer and fine-tunes on them, in the spirit of ADMEDVOICE's synthetic split.

Two ablations isolate the key ingredient. In the first, term spans are relabeled in target spelling but the loss is unmasked, so the model is trained on mixed-script full sentences. In the second, a random-word control relabels an equal number of non-medical Polish words, to check that the effect comes from domain terms and not from code-switching exposure in general.

As an oracle ceiling, we fine-tune on VietMed's labeled training split. The reported reference here is 29.6% WER for XLSR-53-Viet versus a 51.8% baseline in VietMed. For Hindi, the DISPLACE-M winner reports 18.59% tcpWER. That figure is not directly comparable because it includes diarization and about 1,800 hours of Hindi, so we only use it as context.

All trained systems share the same backbone, the same target general hours, the same number of update steps, and the same budget of hyperparameter trials. We count our method as beating the field if it lowers medical-term error on both target sets relative to the best of whole-transliteration, prompting and TTS, with paired-bootstrap significance, and does not raise general target WER by more than half a point.

Why not just…?

Why not just prompt Whisper with the target-language term list? This is a real threat because it is free. The script-steering paper reports that text prompting often fails to enforce the intended output script, especially in small models (Linear Script Representations). That evidence concerns script choice, though, not vocabulary. Our expectation is that prompting will fix terms whose target spelling Whisper already associates with the sound, and will fail on rare or respelled loan terms. We report term error split by how often each term appears in web text. If prompting matches our method across that split, we will conclude that training is unnecessary and say so.

Why not just synthesize target-language medical speech with TTS? This is also a real threat, since ADMEDVOICE shows that synthetic speech helps. Synthetic speech lacks clinical acoustics such as noise, hurried dictation and accents, while our method keeps real clinical audio. We run TTS alone, our method alone, and both together. If TTS alone wins on both targets, the contribution shrinks to the low-resource case where no decent target synthesizer exists. We would then test that case directly by degrading the TTS to small amounts of training data.

Why not just transliterate the whole transcript, as prior work does? That work demonstrated gains only on related pairs and reported degradation for distant ones. Whole-transcript transliteration is our strongest baseline. The loan-term versus translated-term split shows why it should lose: it gives the model target-script strings for Polish words that have no target-language counterpart.

Why not just use a byte-level decoder so the script stops mattering? Whisper's tokenizer is already byte-level. The text evidence (Unknown Script) suggests this removes the problem of representing characters, not the problem of choosing the right spelling. The naive cross-lingual baseline tests exactly this: if byte-level decoding alone were enough, that baseline would already transfer medical terms.

Proposed Experiments

The source domain is ADMEDVOICE natural plus anonymized Polish dictation. The targets are VietMed's test set and DISPLACE-M's development conversations, scored with oracle speaker segmentation so that we measure ASR alone. We report WER (tcpWER for DISPLACE-M, to match the challenge), medical-term error rate under normalized variant-accepting matching, and general-domain WER on FLEURS for both targets. The two targets were chosen to cover two distinct character-set problems. Vietnamese is Latin-based but heavy with diacritics and phonetic respellings. Hindi is a different script with code-switched Latin terms.

There are three decisive checks. First, the loan-term/translated-term split: our method should help mainly on loan terms. Second, the random-word control: relabeling non-domain words should give no medical-term gain. Third, the masked-versus-unmasked ablation: if unmasked mixed-script labels do as well as masked ones, then the masking idea is unnecessary and the simpler variant should be reported as the method. A final probing study tests whether term-identity probes trained on Polish audio transfer to Vietnamese audio after training. That would tie the observed term-error gains to the proposed acoustic-to-spelling mechanism rather than to decoder memorization.