Large speech recognizers built on language models now produce transcripts one word piece at a time. A common way to make them faster is speculative decoding. A small "draft" model guesses several upcoming tokens, and the large model checks all of them in one pass, keeping the guesses it agrees with. When verification is exact, the transcript is identical to what the large model alone would produce, so accuracy cannot change. Speed can. Speed depends on how often the drafter guesses right. A drafter that understands some speakers worse than others therefore makes those speakers wait longer and costs more energy to serve. That gap would not appear in any word error rate table, because the transcripts are unchanged.
We propose to measure this hidden cost in speech for the first time and to separate its two likely sources. The first is acoustic: the small drafter reads voices that are underrepresented in training, such as women in some languages, non-binary speakers, and minority-dialect speakers, less reliably. The second is linguistic: casual, spontaneous speech is out of domain for a drafter trained on read or formal text. The question matters for what we build next. If the gap is mostly linguistic, ordinary domain distillation on conversational data should close it. If it is acoustic, the fix has to give the drafter better access to the voice. We then test a label-free correction that targets whichever source dominates.
Core Claim
We expect speculative decoding in language-model-based speech recognition to impose a latency penalty on underrepresented speakers. We further expect the penalty to be largest when demographic underrepresentation coincides with spontaneous, informal speech. Our main hypothesis is that the acoustic part of the penalty depends on how much of the speech representation the drafter can see. Drafters that reuse the large model's internal states should show smaller demographic gaps than an independent small model that only receives encoder features. A second, testable claim is that up-weighting high-divergence utterances during white-box distillation can shrink the remaining gap without demographic labels at training or inference time, and without changing any transcript.
The closest prior work is The Disparate Impacts of Speculative Decoding. It shows for multilingual text that an under-fit drafter lowers acceptance rates on low-resource languages, and it reports a 13% acceptance gap across MGSM languages. Its correction, s-CDF, focuses drafter updates on the slowest tasks, which requires knowing the groups. Our proposal differs in three ways. The modality is speech, where the drafter's input is an acoustic representation rather than a language identity. The groups are speaker demographics, which are typically unlabeled and cross-cut language. And the design factorially crosses those groups with a read-versus-spontaneous style shift, so acoustic and linguistic causes can be separated rather than confounded.
Research Program
The test system follows SpecASR, which accelerates a Whisper-conditioned Llama-7B or Vicuna-13B recognizer with a TinyLlama drafter. It adds confidence-based early truncation of drafts and reuse of rejected draft tokens, and reports 3.04x–3.79x speedup on LibriSpeech with no change in error rate. That evaluation used read audiobook speech only and gave no speaker breakdown. The paper does note that speedups fall on the harder test-other split, which hints that drafter quality is uneven. We reimplement this setup with greedy exact-match verification, so every speedup we report comes with an unchanged transcript.
The first stage is measurement. The key resource is the Casual Conversations v2 data used in The Balancing Act. It contains both scripted and spontaneous Portuguese recordings from the same speakers, labeled for gender, age, skin tone and region. This allows a within-speaker comparison: the same voice reading versus chatting. Because that dataset is in Portuguese, the target model and drafter will need multilingual capability. For English, we pair crowdsourced read prompts from Common Voice, which carries self-reported gender including an "Other" category, with the conversational podcast and interview audio of FaiST, which carries self-identified race labels.
Several confounds could mimic a demographic effect: speaking rate, utterance length and transcript length all change how many tokens are drafted. We therefore follow FaiST's approach of using mixed-effect regression with speaker random effects and speaking-rate controls. In our case the regression predicts per-utterance mean accepted draft length and wall-clock time per second of audio, rather than word errors. We also report energy per utterance. The motivation is Energy Considerations of Large Language Model Inference, which finds that memory-bound decoding dominates serving energy, so a latency gap is also a cost gap.
The second stage is decomposition, which is the analytic core. For each utterance we compute the drafter–target divergence twice. The first measurement uses the normal audio-conditioned drafter. The second uses a drafter that receives the reference transcript prefix but a degraded acoustic channel, either mean-pooled or replaced with features from a matched majority-group speaker saying the same scripted prompt. The scripted portion of Casual Conversations v2 makes the matched-speaker version possible. If removing speaker-specific acoustics barely changes the demographic gap, the gap is linguistic. If it erases the gap, the gap is acoustic.
We also probe how much gender information the drafter's internal states carry. Twists, Humps, and Pebbles found that in multilingual recognizers, stronger internal gender extractability correlated with smaller female error gaps (ρ = −0.65). It also found that raw pitch and intensity did not explain the gaps. That result shapes our design. It argues against assuming the drafter simply fails on high-pitched voices. It also motivates testing whether drafters that encode speaker characteristics less clearly show larger acceptance gaps. We therefore compare three drafter families of matched drafting cost. The first is an independent TinyLlama reading Whisper features. The second is the same drafter given a projection of the target decoder's hidden states. The third is a set of lightweight prediction heads on the target model's own states, in the style of Medusa as described in the speculative decoding survey.
The third stage is correction. We distill the independent drafter on target logits, using a mix of read and spontaneous audio. Following Training Domain Draft Models for Speculative Decoding, we use offline white-box distillation, which that paper found beats black-box fine-tuning by 2–10% and online distillation by 11–25% in acceptance on text domains. The one change is that each utterance's loss weight increases with its current drafter–target KL divergence, recomputed each epoch. This is a label-free analogue of s-CDF's focus on the slowest groups. High divergence is the measurable symptom of a slow utterance, and it can be computed without knowing who is speaking.
Baselines
The naive reference point is plain autoregressive decoding of the target model. It has no speculation, so its demographic latency gap is whatever arises from differences in transcript length alone. Every speculative method is reported relative to it per subgroup, which tells us how much of any observed gap is added by speculation.
The strongest existing system is SpecASR with its early truncation and draft recycling, run with its original TinyLlama drafter. SpecASR reports 3.04x–3.79x speedup over autoregressive decoding on LibriSpeech and 1.25x–1.84x over prior speculative baselines, with zero error-rate loss. No reported result was found on Lacuna for SpecASR or any speech speculative decoder broken down by speaker gender, race, or speaking style, or evaluated on conversational corpora. We run it on exactly our evaluation sets on the same hardware. Beating it means two things: our corrected drafter shrinks the worst-group-to-best-group ratio of mean accepted length, and it does so without lowering the best group's speedup.
The strongest drafter-adaptation baseline is uniform offline white-box distillation on the same read-plus-spontaneous training audio, with the same steps, learning-rate sweep and data. This is the recipe from Training Domain Draft Models for Speculative Decoding, which reports 82.6% acceptance on function calling and a 25.4% acceptance gain from 4,000 biology samples. It differs from our method only in the per-utterance weighting, so it is the ablation that isolates our key ingredient.
We also run s-CDF from The Disparate Impacts of Speculative Decoding with oracle demographic labels. On MGSM it reports a 20% reduction in acceptance-rate variance across languages. We treat it as an upper reference for what group-aware correction can achieve. Our label-free method succeeds if it recovers most of that reduction.
Finally, Online Speculative Decoding updates the drafter during serving from verification outputs and reports 1.42x–2.17x speedup on text. We run it as the adaptive alternative on a stream that interleaves read and spontaneous utterances in realistic demographic proportions. All drafter variants receive equal training compute, and their per-token drafting cost is equalized within 10%.
Why not just…?
Why not just distill the drafter on conversational speech and call it done? This is the most serious threat. If the gap is mostly linguistic, uniform white-box distillation on spontaneous audio should erase it, and our weighting adds nothing. The decomposition stage answers this directly. If swapping in majority-speaker acoustics leaves the gap unchanged, and the uniformly distilled drafter equalizes subgroups, we will report that domain distillation is sufficient. That would still be a useful finding, because nobody has shown it for speech demographics.
Why not just rebalance the training data by demographic group? The Balancing Act found that naive oversampling sometimes worsened ASR disparities, while synthetic oversampling (SMOTE) helped. That evidence concerns the recognizer, not the drafter. More importantly, rebalancing needs speaker labels that production audio lacks. We include a label-based rebalanced distillation run on Casual Conversations v2. If it matches our label-free weighting, the weighting's value lies only in not needing labels.
Why not just rely on SpecASR's early truncation, which already stops drafting when confidence drops? Truncation can turn low acceptance into short drafts rather than wasted verification. That could make a large acceptance gap a small wall-clock gap. We therefore report acceptance length and wall-clock time separately, with and without truncation. If truncation alone flattens the wall-clock gap, the practical harm is small, and we will say so.
Why not just use self-speculative heads on the target model? If our acoustic hypothesis is right, this may be the simplest fix of all. It is one of the three architectures we compare, not a competitor we argue against.
Proposed Experiments and Decisive Checks
The first check is a sanity condition. Every speculative configuration must give transcripts byte-identical to the target model's greedy output on all evaluation sets. If it does not, the comparison is not lossless.
The first decisive measurement is the subgroup coefficient for mean accepted length in the mixed-effect model, controlling for speaking rate and transcript length. We estimate it for gender, age and skin tone on Casual Conversations v2, gender including "Other" on Common Voice, and race on FaiST. We also estimate its interaction with spontaneous versus scripted style. A significant interaction would confirm that demographic underrepresentation and informal-speech mismatch compound rather than add. A null result on all subgroups, after controls, would itself be the main finding and would end the mitigation line.
The second decisive measurement is the acoustic-swap decomposition on scripted Casual Conversations v2 prompts, which separates the acoustic share of the gap from the linguistic share.
The third is the three-architecture comparison. We test whether drafter access to target decoder states shrinks the acoustic share. We also test whether, across languages and subgroups, gender probe accuracy on drafter states correlates negatively with acceptance gaps, mirroring the error-gap result of Twists, Humps, and Pebbles.
The fourth compares divergence-weighted against uniform distillation. Success requires two results. Our method must cut the worst-to-best subgroup ratio of accepted length relative to uniform distillation. It must also keep overall LibriSpeech speedup within the range SpecASR reports.
As a secondary check, we repeat the measurement with the coarse-grained acceptance rule of Principled Coarse-Grained Acceptance for Speculative Decoding in Speech, applied to its text-to-speech setting on LibriTTS. That method groups acoustically similar codec tokens in the target model's embedding space. If the groups are drawn from a biased embedding space, they may be looser for majority voices. That would show up as group-dependent speedup and group-dependent quality loss in one pass. We would measure the quality loss by an ASR model's word error rate on the synthesized audio.
References
- The Disparate Impacts of Speculative Decoding
- SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding
- The Balancing Act: Unmasking and Alleviating ASR Biases in Portuguese
- FaiST: A Benchmark Dataset for Fairness in Speech Technology
- Energy Considerations of Large Language Model Inference and Efficiency Optimizations
- Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps
- Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
- Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
- Online Speculative Decoding
- Principled Coarse-Grained Acceptance for Speculative Decoding in Speech