Learning Which Speech-Recognition Errors Hurt Patients, Speaker by Speaker

Hospitals now run a large share of clinical documentation through speech recognition: doctors dictate, or an "ambient scribe" listens to the visit, and a draft note appears. These systems are judged by word error rate, the share of words transcribed wrongly, and that metric counts every mistake the same. A dropped "the" costs as much as "hydroxyzine" turning into "hydralazine." Research on accented speech has shown for years that humans do not count errors uniformly. Some mispronunciations barely register while others dominate a listener's impression, and which ones matter depends on the speaker's native language and on the listener's. This proposal carries that idea into clinical speech recognition. The "listener" whose weights we want is the downstream clinical process: the pharmacist, the ordering system, the next clinician who reads the note. We estimate those weights from what actually happened to patients in a hospital's archive.

Concretely, we combine archived dictation audio, the machine draft, the clinician-signed note, and the patient's subsequent record into a retrospective cohort study. A retrospective cohort study analyzes historical records as though they were an experiment, with statistical adjustment standing in for randomization. From this we estimate, for each error type and each speaker native-language group, how much an uncorrected error raises the risk of a downstream medication discrepancy. The result is a harm-weighted error metric and an error-flagging system. We test whether it predicts real downstream problems better than word error rate, better than simply up-weighting medication names, and better than asking clinicians to rate severity.

Core Claim

We make two claims. First, error weights for clinical speech recognition can and should be estimated from downstream consequences rather than assumed. Doing so credibly requires the confounding control of observational epidemiology, because the notes that contain errors are not a random sample. They come from rushed clinics, complex patients, and particular speakers. Second, these weights are language-specific. The native language of the person speaking determines which sound confusions occur, and so which drug-name and dose confusions reach the chart. A metric or flagging model conditioned on speaker native language should therefore beat one pooled across speakers. The new element is the combination: outcome-derived weights, estimated with a within-speaker cohort design and conditioned on native language. Neither the speech-assessment literature nor the clinical speech recognition literature in our file does this.

What Prior Work Gives Us

Three findings from pronunciation research motivate the language-specific half. EpaDB annotated 3,200 utterances from 50 Argentine Spanish speakers and found that error frequency and perceptual impact come apart. The dark l was mispronounced 84.9% of the time with little effect on nativeness ratings, while errors on schwa, [V] and [AE] correlated with ratings at 0.88–0.93. Using human perception for automatic accent assessment supplies the method template. Its P-WFST system aligns decoded and canonical phones, then trains a maximum-entropy model, a simple weighted classifier, on human accent ratings, so that each error type receives a learned penalty. We reuse that structure but replace human perceptual ratings with adjusted downstream clinical outcomes as the training signal. What influences the foreign accent strength? and Perception of disfluency: language differences and listener bias show that the weights depend on the listener's language as well as the speaker's. Polish listeners penalized phonological errors more than grammatical ones, German listeners did not, and listeners under-detect disfluencies in unfamiliar languages. These results justify conditioning weights on language rather than pooling.

Two results constrain the design. Phone-Level Pronunciation Scoring for Spanish Speakers Learning English Using a GOP-DNN System found that a native-trained acoustic model's goodness-of-pronunciation score was reliable on some phones (equal error rate 0.12 on [EY]) and worse than chance on others (0.58 on [DH]). It also found that duration normalization erases vowel-length contrasts. We therefore derive confusion profiles from recognizer output, not from goodness-of-pronunciation scores. L2-ARCTIC reports that a baseline goodness-of-pronunciation detector reached only 0.29 precision and recall on substitutions, which supports the same choice.

On the clinical side, Speech Recognition for Medical Conversations is the closest engineering work. It reports unweighted word error rate on 14,000 hours of proprietary visits: 18.3% for the attention model, 17.4% for doctors versus 23.3% for patients, and 98.2% recall on medication names. It does not ask which errors matter or follow any patient forward. MEDEC shows that clinical text errors differ by category (diagnosis, pharmacotherapy, and others) and that language models detect them imperfectly, but it works on text with injected errors and has no audio. The epidemiological machinery comes from propensity-adjusted cohorts such as Tewari et al., where crude comparisons were driven by who received which treatment. It also draws on the hospital-nested multilevel models of Wen et al., who found that outcomes varied systematically across 375 hospitals.

Research Program

The program has two phases. The first uses only public data and makes a prediction that the second tests.

In Phase 1 we build native-language-specific confusion profiles. We run a strong English recognizer on L2-ARCTIC (Hindi, Korean, Mandarin, Spanish and Arabic speakers) and on EpaDB (Spanish), align output to reference phones, and estimate a phone confusion matrix per native language. We then project each matrix onto a pronunciation lexicon of drug names, dose words and number words. For each native language this gives a predicted list of the name pairs most likely to be confused, for example pairs that differ only in a vowel that Spanish speakers merge or in a voicing contrast. L2-ARCTIC has only two speakers per language, so these matrices are noisy. We report bootstrap intervals over speakers and treat Spanish, backed by EpaDB's 50 speakers, as the best-powered case.

In Phase 2 we run a retrospective cohort study inside a partner health system that has archived dictation audio, machine drafts, signed notes and linked records. We will seek approval from the ethics review board for the use of these records. The unit is an error: a token where the machine draft differs from what was actually said. Ground truth comes from a double-annotated audited sample, and silver labels come from the signed note. We track whether the error survived proofreading into the signed note. The primary outcome is a proximal, well-recorded event within 72 hours: a pharmacist intervention, an order modification or cancellation, or a medication-reconciliation discrepancy tied to the note's content. Thirty-day readmission is secondary and exploratory, because a transcription error is a weak lever on it.

The key design choice is to compare within speakers. The same clinician dictates hundreds of notes, some with surviving errors of type $k$ and some without. Clinician fixed effects then absorb accent, specialty, experience and writing habits, which are the largest confounders. Note-level propensity adjustment handles the rest: patient complexity, note length, time of day, clinic load and the drug classes mentioned. Hospital-level random effects follow Wen et al. The harm weight for error type $k$ among speakers of native language $\ell$ is the adjusted risk difference

$$w_{k,\ell} = \Pr(Y=1 \mid E_k=1, X, c, \ell) - \Pr(Y=1 \mid E_k=0, X, c, \ell),$$

where $Y$ is the downstream event, $E_k$ indicates a surviving error of type $k$ in the note, $X$ are note-level covariates, and $c$ is the dictating clinician. Error types are defined at two levels: a clinical category, using MEDEC's scheme plus dose and laterality, crossed with a phonetic mechanism such as vowel merger, voicing or deletion. Speaker native language comes from a consented clinician survey, not from accent inference on audio, both for privacy reasons and because inferred labels would add measurement error. The weights then feed two products: a harm-weighted error rate for evaluating recognizers, and a flagging model that ranks draft tokens for review, trained in the maximum-entropy style of P-WFST.

One hypothesis from the perception literature is tested directly. Wesołek et al. found in The influence of L2 accent strength and different error types on personality trait ratings that individual errors lose weight as overall accent strength rises. Where human editors correct drafts while listening to the audio, we test whether error survival rises with the dictating speaker's accent strength, measured by an automatic accent score, holding recognizer error rate fixed. A positive result would locate part of the risk in the human review step rather than the machine.

Baselines

The naive reference point is unweighted word error rate, computed on the same audited notes and used as a note-level score to predict the downstream event. It is what Chiu et al. report, 18.3% word error rate for their best model, and it is the metric vendors publish. On the question we care about, predicting harm, we found no reported result on Lacuna for word error rate as a predictor of downstream clinical events. We will establish the number ourselves.

The strongest simple competitor is clinical-entity-weighted error rate. It counts only errors on medication names, doses, laterality and diagnoses, or gives them a fixed large weight, following MEDEC's categories. This baseline captures most of what an obvious engineer would do, and if outcome-derived weights add nothing beyond it, the cohort machinery is not worth its cost. Its closest reported figure is the 98.2% medication-name recall in Speech Recognition for Medical Conversations. That is a recall, not a harm prediction, so we will measure the baseline's predictive performance directly.

The third baseline is clinician-elicited severity. A panel of pharmacists and physicians rates a sample of errors, and a P-WFST-style maximum-entropy model is trained on those ratings, reproducing Using human perception for automatic accent assessment with clinical raters in place of accent raters. The reference numbers from that paper are a speaker-level correlation of 0.86 versus 0.75 for goodness-of-pronunciation on Mandarin speakers. That was measured against perceptual ratings, not clinical outcomes.

For the flagging task we add a text-only language-model detector run on the draft note, the approach MEDEC evaluates. There Claude 3.5 Sonnet reached 70.16% detection accuracy, against 71.6–79.6% for physicians.

All systems will be fit on the same training months, tuned with the same budget of held-out validation months, and evaluated on later held-out months and a held-out hospital. Two ablations isolate the proposal's ingredients: pooled weights without native-language conditioning, and weights estimated without clinician fixed effects or propensity adjustment. Beating a baseline means a statistically reliable gain in held-out discrimination (area under the curve for the 72-hour event) and in precision among the top 1% of flagged tokens, with intervals from clustered bootstrap over clinicians.

Why not just…?

Why not just up-weight medication and dose errors and skip the cohort study? This is the real threat, and the entity-weighted baseline exists to test it. Our reason to expect a gain is that entity weighting treats all medication errors alike. Harm probably depends on whether the substituted word is itself a plausible drug in context: a nonsense string gets caught, while a real, confusable drug slips through. It may also depend on the speaker's native language, through which confusions occur. If held-out prediction shows no gain over entity weighting, we will report that outcome-derived weights collapse to entity weights and recommend the simpler metric.

Why not just ask clinicians how severe each error is? Expert ratings measure how bad an error looks, not whether it propagates. Proofreading, pharmacist checks and order-entry constraints catch some alarming-looking errors while missing mild-looking ones. Rater agreement on clinical errors is also limited, about 69% in MEDEC. The elicited-severity baseline answers this directly. If it matches outcome weights, the expensive part of the program is unnecessary.

Why not just correlate note-level error rate with outcomes across the archive? Notes with more errors come from rushed clinics, complex patients and particular speakers, so a crude correlation would mostly reflect those factors. This is the confounding by indication documented in Tewari et al.. The unadjusted ablation quantifies how far crude weights diverge from within-speaker adjusted ones.

Why not just let a language model clean up the draft? It has no access to the audio, so it cannot tell a correctly transcribed unusual drug from a plausible substitution. Detection on MEDEC's real clinical notes also remains below physician level. We include it as a baseline. We also test a combination that feeds the language model the flagged tokens and the native-language-specific confusion candidates. If the language model alone matches our flagger, that is a useful and publishable negative result.

Decisive Experiments

The first check connects the two phases. Among observed drug-name substitutions in the archive, pairs predicted by the Phase 1 confusion matrix for the speaker's own native language should be over-represented relative to pairs predicted by other languages' matrices. We will test this with a permutation test over language labels. If this enrichment is absent, the language-specific half of the claim fails at the mechanism level, whatever the prediction numbers show.

The second check is held-out prediction of 72-hour medication events: the outcome-weighted metric and flagger against the four baselines and two ablations, on later months and an unseen hospital.

The third check is a negative control. Surviving errors in note sections that plausibly cannot affect orders, such as social history filler, should receive weights near zero. Non-zero weights there would indicate residual confounding.

The fourth check is the accent-strength test of error survival in audio-assisted editing, the clinical analogue of the masking effect.

The main limitation is data access. No public corpus in our file links clinical audio to outcomes, and the Chiu et al. corpus is private. Phase 1 and every baseline definition are fully public so that any hospital with archived dictation can run Phase 2.