Speech recognition systems increasingly hand their rough transcripts to a large language model (LLM) that "fixes" them. The fixer can now do more than choose among the recognizer's guesses. It can write words the recognizer never proposed, and on read news speech this beats the best possible pick from the recognizer's own top-10 list. That ability helps most where the audio is hardest to understand. Our concern is that the audio is hardest to understand for speakers the recognizer already serves poorly: non-native, accented, child, and elderly speakers. For those speakers the fixer has the least acoustic evidence to go on, so it relies most on its own expectations of what people usually say. Those expectations include societal stereotypes.
The proposal is to measure what the corrector changes in the meaning of what each speaker said, not only in the words. We break both the true transcript and the corrected transcript into small, self-contained factual statements, following the method used to score LLM factuality. We then count, for each speaker group, how many statements were kept, dropped, distorted, or invented. Our central prediction is that generative correction can shrink the word-error gap between speaker groups while widening the gap in invented and distorted statements. We also predict that the distortions lean toward whatever the text-only language model finds most likely, stereotypes included. If this holds, fairness evaluations that track only word error rates will report progress precisely where the harm is growing.
Core Claim
Word error rate (WER), the fraction of words substituted, deleted, or inserted, treats "pacemaker" → "taste maker" as one error, the same as "uh" → "um". Prior work on speech fairness measures WER parity across demographic groups. Quantifying Bias in Automatic Speech Recognition found 55.9% WER for non-native versus 26.1% for native Dutch speakers. ASR-FAIRBENCH combines WER with a regression-based fairness score. Both motivate the gap, and both stop at lexical errors on a fixed recognizer. They never examine a generative corrector, and they never ask whether errors change what the speaker asserted.
Work on generative error correction supplies the method we audit. Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting shows a low-rank-adapted (LoRA) LLM reaching 2.11% WER on WSJ, against an N-best oracle of 9.78%. It evaluates only WER and never stratifies by speaker. ContextASR-Bench documents a related "hallucination trap": audio language models transcribe entities from the context prompt that were never spoken. It measures this at the entity level only, and not by speaker group.
Our claim is specific. The amount of content a corrector invents is governed by how weak the acoustic evidence is. Acoustic weakness is distributed unequally across demographic groups, so the corrector converts an acoustic disparity into a semantic one. The invented content is pulled toward high-prior completions. FairBelief shows such completions carry stereotypes at top-1 likelihood. Uncovering Stereotypes in Large Language Models shows alignment guardrails weaken as tasks get more complex.
Measuring Meaning, Statement by Statement
The audit reuses the decomposition-and-verification design of FActScore. That method splits long text into atomic facts, resolves pronouns so each fact stands alone, and checks each fact against evidence. Its automated version reaches under 2% error relative to human annotators on biographies. We apply it in both directions between a human reference transcript and a system transcript.
First, each atomic statement from the reference is checked against the system transcript with a natural language inference (NLI) model, which judges whether one text entails, contradicts, or is neutral toward another. An entailed statement is kept, a contradicted one is distorted, and a neutral one is dropped. Second, each atomic statement from the system transcript is checked against the reference. Any statement not entailed by the reference is invented.
This yields claim recall (the share of what was said that survives) and a fabrication rate (the share of what the transcript asserts that was never said). The precision-only design of FActScore is a known limitation of that line of work. The two-way check addresses it because the reference transcript is a natural ground truth for completeness.
For every distorted or invented statement, we record a prior pull score. This is the log-probability, under the corrector's base LLM with no audio-derived input, of the substituted content minus that of the true content, given the surrounding reference context. A positive prior pull means the corrector moved toward what its language model expected. A subset of statements touches protected attributes: gendered references, nationality, religion, family roles, and occupation–person pairings. On those, we further label whether the distortion moves toward or away from the stereotype-congruent value.
This connects to the directional distinction in Directional Bias Amplification, which separates attribute-to-task effects from task-to-attribute effects. In our setting, "speaker group raises fabrication" is the attribute-to-task path. "Topic context rewrites the attributes described in the speech" is the task-to-attribute path. That paper's "masking paradox" applies here too: suppressing one pathway can inflate the other. That paper also shows that threshold calibration alone can collapse an apparent amplification from 0.040 to 0.001. We therefore report every disparity relative to group base rates of the relevant content, not as raw counts.
Research Program
The work has three parts.
The first part builds and validates the audit. Spontaneous speech contains disfluencies and fragments, which WER we are and WER we think we are shows degrade downstream processing. Decomposition noise is also a known weakness of atomic evaluation. So before any fairness claim, we hand-annotate a few hundred utterances per corpus for statement status. We then measure agreement between the automated audit and humans, separately for each speaker group. If the decomposer is itself less accurate on non-native speech, any downstream disparity is confounded. This per-group agreement check is a required gate, not an afterthought.
The second part runs a controlled correction study. We hold a first-pass recognizer fixed: Whisper-Large-v3, plus the Dutch Corpus Gesproken Nederlands (CGN) trained system where we can reproduce it. We vary only the corrector, along one axis: how far it may depart from the acoustic hypotheses. This ranges from no correction, to choosing within the N-best list, to free generation with task-activating prompting, to free generation with LoRA tuning, and finally to free generation with domain context in the prompt. We do this on corpora with speaker metadata: Jasmin-CGN (Dutch children, teenagers, older adults, non-native and Flemish speakers) and the Fair-Speech subset used by ASR-FAIRBENCH (U.S. speakers stratified by age, gender, ethnicity, and first language). The domain-mismatch stress comes from ContextASR-Bench's entity-dense English dialogue. That benchmark lacks speaker demographics, so it serves to measure the context-hallucination path rather than group gaps.
The third part is a counterfactual probe that isolates prior pull from acoustic difficulty. We write short scripts that contain counter-stereotypical statements (for example, a male nurse, or a female engineer referred to with pronouns). We have them read by native and non-native speakers, or render them with voice conversion as a cheaper secondary condition. We then measure how often each corrector flips the statement to the stereotype-congruent version. Because the text is identical across speakers, any difference in flip rate between groups is attributable to the speech signal interacting with the corrector's prior.
The constructive outcome, if the effect is real, is a correction policy gated by acoustic evidence. Free generation is permitted only on spans where first-pass word confidence is low and the proposed rewrite is phonetically close to the hypothesis. Rewrites with high prior pull on attribute-bearing statements are rejected. We treat this as a test of the mechanism, not as the headline contribution.
Baselines
The naive reference point is no correction: the first-pass Whisper or CGN output, audited at the claim level by speaker group. Every other system is compared against this, because the question is whether correction adds semantic disparity beyond what the recognizer already causes. No claim-level results on these corpora are reported on Lacuna. The closest existing figures are lexical. Jasmin-CGN has 55.9% non-native versus 26.1% native WER, per Quantifying Bias in Automatic Speech Recognition. On Fair-Speech, whisper-medium has average WER 0.06 and FAAS 29.41, per ASR-FAIRBENCH.
The strongest existing correction method is LoRA-tuned generative error correction from Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting. It reports 2.11% WER on WSJ, against an N-best oracle of 9.78%, and 1.69% on ATIS. It is the strongest competitor because it achieves its gains exactly by writing beyond the acoustic hypotheses, which is the behavior we hypothesize is harmful. We will reproduce it on WSJ to confirm our implementation lands near that figure before trusting it elsewhere. We then tune it on the same in-domain training split, with the same LoRA rank and step budget, for every corpus. "Beating" it means the gated policy matches its WER within a pre-registered margin while reducing both the between-group fabrication gap and attribute-flip rates.
The key ablation is N-best-constrained correction: the same LLM, prompt, and tuning budget, but required to output one of the recognizer's top-10 hypotheses. This isolates the ingredient we care about, freedom to generate beyond the acoustic evidence. A second ablation removes domain context from the prompt, to separate prior pull from the context-induced hallucination described in ContextASR-Bench. Finally, a sequence-to-sequence corrector in the style of ASR Error Correction and Domain Adaptation Using Machine Translation serves as a smaller-model reference. That paper reports 41.0% → 34.1% WER for Google ASR on medical dialogue. Its proprietary data is unavailable to us, so we train only its architecture on our splits.
Why not just…?
Why not just stratify WER or named-entity WER by speaker group, as ASR-FAIRBENCH does? This is the most serious alternative. If claim-level disparities simply track word-level ones, the audit adds cost without insight. The decisive test is built into the analysis. We regress each group's fabrication and distortion rates on its WER, using matched utterances, and ask whether non-native and child speech show excess semantic damage beyond what their WER predicts. We also test whether free generation narrows the WER gap while the fabrication gap widens, a divergence WER cannot show by construction. Entity WER from ContextASR-Bench misses relational and attribute statements, such as who did what to whom, which is where stereotype flips live. If the excess is zero, we will report that stratified WER is sufficient, and that is a useful negative result.
Why not just forbid the corrector from leaving the N-best list? This is a real threat to the significance of the work, because it would remove invention almost entirely. The cost is the gain that motivates generative correction in the first place. The reported WSJ gap is 9.78% oracle versus 2.11% generated, so constraining to the list forfeits most of the benefit. The N-best ablation measures this trade-off directly on each speaker group. If the constrained corrector keeps nearly all the WER improvement for disadvantaged groups at negligible fabrication, the gated policy is unnecessary, and we would say so.
Why not just fine-tune the recognizer on accented, in-domain speech? It is often expensive, and the evidence is mixed. ASR-FAIRBENCH reports that fine-tuned Wav2Vec and HuBERT models lag in fairness. Quantifying Bias in Automatic Speech Recognition finds that higher non-native proficiency does not close the gap. More to the point, deployed pipelines are already adding LLM correctors on top of whatever recognizer exists, and fine-tuning the recognizer does not tell us what the corrector does to meaning. Where a Jasmin-adapted recognizer is feasible, we include it to test whether stronger acoustics shrink prior pull, as our mechanism predicts.
Why not just rely on the corrector's safety alignment? Correction prompts are not the explicit stereotype questions that alignment is tuned on. FairBelief finds hurtful completions at peak likelihood in an instruction-tuned model, with Vicuna 7B scoring 0.257 on HONEST. The counterfactual probe tests this directly by comparing aligned and base correctors on identical scripts.
Proposed Experiments and Decisive Checks
The experiments run on Jasmin-CGN, the Fair-Speech subset, ContextASR-Bench English dialogue, WSJ (for reproduction), and the newly recorded counterfactual scripts. The primary metrics are claim recall, fabrication rate, distortion rate, prior pull, and stereotype-flip rate on attribute-bearing statements. Each is reported per speaker group and corrected for group base rates. WER and FAAS are reported alongside for continuity with prior work.
Three results would decide the question. First, the audit must agree with humans equally well across speaker groups; if it does not, we stop and fix the decomposer. Second, the fabrication gap under free generation must exceed both the no-correction gap and the N-best-constrained gap, beyond what WER differences predict. Third, on the counterfactual scripts, the flip rate toward the stereotype must be higher for non-native and voice-converted speech than for native speech with the same text, and prior pull should be positive on flipped statements. If all three hold, generative correction is a channel through which acoustic inequity becomes semantic and stereotyped error. If the second fails, the honest conclusion is that stratified WER already tells the fairness story for these correctors.
References
- Quantifying Bias in Automatic Speech Recognition
- ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems
- Generative Speech Recognition Error Correction With Large Language Models and Task-Activating Prompting
- ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark
- FairBelief - Assessing Harmful Beliefs in Language Models
- Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Directional Bias Amplification
- WER we are and WER we think we are
- ASR Error Correction and Domain Adaptation Using Machine Translation