Speech recognition benchmarks for accented speech now routinely report that some accent groups fare much worse than others. LAHAJA, a Hindi benchmark, reports about 20.5% word error rate (WER) for Assamese speakers against about 10.5% for Urdu and Sindhi speakers. Svarah, an Indian English benchmark, singles out North-Eastern speakers as particularly poorly served. These claims rest on roughly six or seven speakers per language background, and neither benchmark reports an uncertainty. Experimental physics has a long habit of asking a question these benchmarks skip: given how few independent events you observed, what is the smallest effect you could have detected at all? This proposal brings that question to accent benchmarks.
The core idea is simple. In an accent benchmark the scarce "events" are speakers, not utterances, because all of one speaker's errors rise and fall together. We will build a sensitivity calculus for multi-accent ASR evaluation. It gives the minimum detectable WER gap as a function of speakers per accent group and utterances per speaker. It corrects claims about the "worst accent" for having searched across many groups. It turns both into a rule for how to allocate a fixed recording budget. We then use it to audit the published disparity claims on Svarah, LAHAJA, and the Dialects of English Archive study, and check the audit against an empirical test that needs no modeling assumptions: do group rankings replicate across disjoint halves of the speakers?
Core Claim
Our central claim is that per-accent WER differences in current multi-accent benchmarks sit in a counting-limited regime. In this regime the precision of a group's WER is capped by the number of speakers in the group, and adding more utterances per speaker barely helps. Under a simple two-level model, a group's mean WER has variance
$$\mathrm{Var}(\hat W_g) \approx \frac{\sigma_s^2}{K} + \frac{\sigma_e^2}{K m},$$
where $K$ is the number of speakers in group $g$, $m$ the number of words scored per speaker, $\sigma_s^2$ the between-speaker variance of true WER within the group, and $\sigma_e^2$ the within-speaker word-level error variance. The minimum detectable gap between two groups at significance $\alpha$ and power $1-\beta$ is then roughly $(z_{1-\alpha/2}+z_{1-\beta})\sqrt{2\,\mathrm{Var}(\hat W_g)}$. Once $m$ is large, this gap stops shrinking and floors at a value proportional to $\sigma_s/\sqrt{K}$. That floor is the same $1/\sqrt{N}$ counting limit that governs rare-event searches, with speakers playing the role of events.
The formula itself is textbook. What is new is three things. First, we estimate $\sigma_s$ and $\sigma_e$ per accent group on real benchmarks. Second, we report which published gaps clear the floor and which do not, after correcting for the number of groups searched. Third, we validate those verdicts against split-half replication.
Relation to Prior Work
The closest work is Statistical Testing on ASR Performance via Blockwise Bootstrap. It shows that resampling utterances as if they were independent badly underestimates WER variance: at intra-block correlation 0.4, ordinary bootstrap 95% intervals covered the truth 41.2% of the time against 95.9% for blockwise resampling. It also finds that honest intervals are 1.5× to 2.0× wider on conversational and AMI meeting data. We take its central lesson as the starting point: the speaker is the unit of independence. We differ in three ways.
First, that paper compares two systems on one pooled test set with hundreds of blocks. We compare groups of five to ten speakers each, where resampling within a group has too few blocks to be reliable. Second, it does not treat speakers nested inside accent groups. Third, it does not address minimum sample sizes or the multiplicity of searching across 19 groups.
The benchmarks motivate the work by supplying the gap. Svarah evaluates eight systems on 117 speakers across 19 languages, about 9.6 hours in total, and reports point estimates only. LAHAJA does the same with 132 speakers across 19 language backgrounds. Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions does run ANOVA (analysis of variance, a classical test for differences between group means) on 39 talkers. It finds a dialect effect for YouTube (F[3,35]=3.45, p<0.05) but not for Bing (F[3,32]=1.6, p=0.21), and the authors themselves leave open whether Bing's null reflects parity or insufficient power. Bing's much larger spread (σ=0.18 against 0.07) makes the second reading plausible, and our calculus can say which.
The physics papers constrain and inspire the method rather than supply it. The Telescope Array anisotropy study (INDICATIONS OF INTERMEDIATE-SCALE ANISOTROPY...) shows how searching many sky regions erodes the significance of the hottest one after a trial-factor correction. We apply the same correction to "the worst-served accent" chosen from 19 candidates. The psi(2S) branching-fraction result, that roughly 2σ fluctuations routinely fail to survive more data, and the TRAPPIST survey finding that 40 targets yield only broad bounds both motivate reporting sensitivity limits and upper bounds rather than point estimates alone.
Research Program
We first obtain per-word, per-utterance error outputs on Svarah and LAHAJA by running openly available systems ourselves: Whisper base, medium and large, plus wav2vec2-family models on Svarah, and MMS and Whisper on LAHAJA. We also include any released LAHAJA checkpoints. Commercial APIs are optional and are not needed for the statistical claims.
We then fit a two-level model per system. Word errors are counted per utterance with a beta-binomial likelihood. Each speaker gets a random effect, nested in a fixed or partially pooled effect for the accent group. This model is fit directly on word-level counts rather than on speaker-averaged WER, which is the speech analog of the unbinned likelihood fits used in rare-event physics. It weights speakers by how much speech they contributed and does not assume WER is normally distributed.
From the fit we report three things. We give per-group intervals. We give the minimum detectable gap at the observed $K$ and $m$, both with no multiplicity correction and with a trial-factor correction calibrated by simulating the maximum gap among 19 groups under a null of no group effect. Finally, we give a design curve showing how many speakers per group are needed to detect gaps of 2, 5 and 10 WER points.
Finally, we answer the Bing-versus-YouTube question from reported statistics alone. Using the published standard deviations and group sizes, we compute the power that study had to detect dialect gaps of various sizes. Where the archive audio is available, we rerun current open systems on the "Comma Gets a Cure" recordings with the speaker-level model.
Baselines
The naive reference point is what the benchmarks do now: a raw per-group WER point estimate with no uncertainty, read as a disparity whenever two groups differ. Against this reference, success means showing which published differences are and are not distinguishable from speaker sampling noise, and confirming that verdict by split-half replication.
The first statistical baseline is the ordinary bootstrap over utterances within each group. It is what most practitioners would reach for. We run it on the same model outputs with 10,000 resamples. We expect it to be overconfident, consistent with the 41.2% coverage reported in Statistical Testing on ASR Performance via Blockwise Bootstrap.
The strongest existing method is that paper's blockwise bootstrap with speakers as blocks, applied separately within each accent group. For the gap between two groups, speakers are resampled independently in each. This is the real competitor because it already respects the speaker as the unit of independence. It is run on identical outputs, with the same resample count and the same nominal level. To beat it, our hierarchical intervals must reach close to 95% coverage in simulation at five to eight speakers per group, a regime where we expect blockwise coverage to fall below nominal because there are so few blocks. They must also match its coverage in the many-speaker regime where it is known to work. If blockwise speaker resampling already achieves nominal coverage at small $K$, we will say so and recommend it, and our contribution reduces to the audit and the design rule.
Speaker-averaged one-way ANOVA, as used in the Dialects of English Archive study, is the classical-phonetics baseline. We compare its power and coverage under matched simulations.
One ablation isolates the key ingredient. We fit the same mixed model but drop the speaker random effect, which makes it equivalent to treating words as independent within a group. If this version matches the full model on coverage, speaker clustering is not what drives our conclusions.
Why not just…?
Why not just run the blockwise bootstrap by speaker and report intervals? This is a real threat, and it may be most of the answer for large groups. With six speakers in a group, however, there are only a handful of distinct resamples worth the name. The interval then reflects those six speakers' idiosyncrasies more than the population, and the blockwise paper never tested this regime. Our simulation sweep over speakers per group directly measures whether blockwise coverage holds there. We will adopt whichever method attains nominal coverage. The trial-factor correction and the speaker-allocation rule are needed either way, and the bootstrap alone supplies neither.
Why not just fit a standard mixed-effects regression and stop? Our estimator essentially is one, and we do not claim it as new. The gap in the literature is not the absence of the tool. It is that no one has used it to state the sensitivity floor of existing accent benchmarks, to correct "worst accent" claims for searching across 19 groups, or to check those verdicts against replication. A mixed model fit once, without the null simulations for the maximum gap and without the design curves, would not tell a benchmark builder whether to record more speakers or more sentences per speaker.
Why not just collect much more data? For test sets that is indeed the remedy, and the question is how to collect it. Our variance decomposition predicts that extra utterances from the same few Assamese speakers add little precision once the speaker term dominates, while extra speakers add a lot. Note that LAHAJA's observation is about training data: 2,285 hours cut WER only 2.4 points relative to its 1,509-hour model while minority-group WER stayed high. That bears on model improvement, not on evaluation precision, so we do not lean on it here. The design curve converts "collect more" into a concrete number of speakers per group for a target detectable gap.
Proposed Experiments
The first decisive check is a coverage simulation calibrated to real data. We fit $\sigma_s$ and $\sigma_e$ per group on Svarah and LAHAJA. We then simulate benchmarks with 3 to 30 speakers per group and realistic utterance counts, plant known gaps between groups, and measure interval coverage and test power for all baselines. Coverage below 90% at small group sizes for any method rules that method out for accent audits.
The second check is empirical and assumption-light. We repeatedly split each benchmark's speakers within every group into disjoint halves and compute group WERs on each half. We then measure how often the sign of each pairwise gap, and the identity of the worst group, replicate. Our model predicts a replication probability for each gap from its estimated size and the half-sample sensitivity. Agreement between predicted and observed replication rates validates the calculus, and disagreement falsifies it.
The third experiment is the audit itself. For every system we report which pairwise gaps exceed the trial-corrected minimum detectable gap. The test case is LAHAJA's Assamese versus Urdu/Sindhi contrast of 20.5% against 10.5%. Our hypothesis is that gaps of this size survive while many smaller inter-group differences do not, though the data could show otherwise. As a point of reference, LAHAJA's best reported aggregate is 14.3% WER for its Conformer-L model M3, against 16.7% to 34.4% for MMS, Whisper V3, Azure and Google Chirp. We will also test whether system rankings within each accent group are distinguishable, since several of LAHAJA's reported model-to-model improvements are only one to three points.
For the Dialects of English Archive study, we report the minimum detectable dialect gap for Bing given σ=0.18 and eight to twelve talkers per dialect. If that gap exceeds the YouTube effect the study did detect, Bing's non-significant result is uninformative rather than evidence of parity.
The final deliverable is a released, versioned analysis that any accent benchmark can run on its own outputs, together with a speaker-allocation table for the next round of corpus collection.
References
- Statistical Testing on ASR Performance via Blockwise Bootstrap
- Svarah: Evaluating English ASR Systems on Indian Accents
- LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems
- Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions
- INDICATIONS OF INTERMEDIATE-SCALE ANISOTROPY OF COSMIC RAYS WITH ENERGY GREATER THAN 57 EeV IN THE NORTHERN SKY MEASURED WITH THE SURFACE DETECTOR OF THE TELESCOPE ARRAY EXPERIMENT
- Branching fractions for<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mi>ψ</mml:mi><mml:mn/><mml:mo>(</mml:mo><mml:mn>2</mml:mn><mml:mi>S</mml:mi><mml:mo>)</mml:mo><mml:mn/><mml:mo>→</mml:mo><mml:mi>γ</mml:mi><mml:mrow><mml:msup><mml:mrow><mml:mi>η</mml:mi></mml:mrow><mml:mrow><mml:mo>′</mml:mo></mml:mrow></mml:msup></mml:mrow></mml:math>and γη
- Global analysis of the TRAPPIST Ultra-Cool Dwarf Transit Survey