An Error Budget for Accented Speech Recognition: Deciding Which Accent Data to Record

Speech recognizers trained mostly on standard dialects make far more mistakes on accented speech. One reported system's word error rate climbs from 18.7% on Canadian English to 49.1% on Indian English. The usual remedy is to collect more accented data, but accent coverage is the scarce resource. Archives are small and dominated by read speech, and every new accent needs new speakers, new recordings and new transcripts. This proposal borrows a habit from particle physics. Physicists predicting the muon's magnetic moment cannot compute the strong-force contribution directly. Instead they write it as a weighted sum of measured collision rates over energy and over separate final-state "channels." They track which energy region and which channel contributes most to the final uncertainty, and spend beam time there. They also measure each signal against a well-understood reference process recorded under identical conditions, so that shared measurement errors cancel.

We propose the same accounting for accented speech. First, split an accent's excess error into "channels," meaning the specific sound confusions a speaker's native language induces (for example, merging two English vowels). Second, weight each channel by how often it actually changes a word in the target text. Third, estimate each channel's excess error from a small probe set with a double ratio against accent-stable control sounds, which cancels speaker and microphone effects. Fourth, spend a fixed recording budget on the prompts that cover the channels with the largest weighted error. The claim is testable in a simple way. Under a fixed number of recorded hours, budget-guided collection should reduce accented word error more than uniform or confidence-based collection, and the budget's ranking of channels should predict which additions actually help.

Core Claim

Two papers set up the physics side. In the muon calculation, the leading strong-force term is an integral of the hadron production ratio $R(s)$ over squared collision energy $s$, with a kernel that weights low energies heavily. Precision work therefore concentrates on the two-pion channel near the $\rho$ resonance and on dense energy sampling. F. De Mori et al. (2006) measured 68 points in a narrow window specifically to control interpolation error. B. Aubert et al. (2009) measured $\pi^+\pi^-(\gamma)$ as a ratio to the purely electromagnetic $\mu^+\mu^-(\gamma)$ process in the same data. This cancelled luminosity, tracking and acceptance errors. We reuse two of these ideas as method: the kernel-weighted, channel-decomposed error budget, and the same-sample reference ratio.

We do not reuse the dispersion relation itself. Speech representations obey no analyticity or unitarity constraint that would justify a dispersion-style bound. We therefore reject that candidate bridge as metaphor.

The speech-side quantity is the predicted word-error gap of target accent $a$:

$$\Delta_a = \sum_c w(c)\, e_a(c).$$

Here $c$ ranges over phone-confusion channels, meaning a phone in a given context realized as another phone. The weight $w(c)$ is the frequency of channel $c$ in the target text, multiplied by the fraction of its occurrences where the confusion would change the word. This is the "kernel," computed from text and a pronunciation lexicon alone. The term $e_a(c)$ is the recognizer's excess confusion rate on that channel for accent $a$.

We estimate $e_a(c)$ with a double ratio:

$$\hat e_a(c) = \frac{\epsilon_a(c)/\epsilon_a(\text{ctrl})}{\epsilon_{\text{ref}}(c)/\epsilon_{\text{ref}}(\text{ctrl})}.$$

In this formula, $\epsilon$ is the measured phone error rate, "ref" is the standard dialect, and "ctrl" is a pool of control phones whose error rate varies least across seen accents. Errors that scale all phones together, such as noise, microphone and speaking rate, cancel in the inner ratio. This is the analogue of luminosity cancelling in the BABAR measurement. What remains is accent-specific.

The novelty is not a new recognizer architecture. It is a measurable, falsifiable model of where an accent's errors come from, used to decide what to record.

Research Program

The work has three stages.

Stage 1: validate the channel estimator against ground truth. We use L2-ARCTIC, which annotates 5,199 phone substitutions, 1,048 deletions and 497 additions across 1,499 utterances from speakers of Hindi, Korean, Mandarin, Spanish and Arabic. The test is whether $\hat e_a(c)$, computed from a recognizer's outputs on a small probe set, ranks the human-annotated substitution channels correctly. We compare this to two simpler estimates: raw per-phone error with no ratio, and confusion matrices built in the manner of Bouselmi et al. and Shao et al.. Those works injected native-versus-non-native confusion statistics into decoding. We use similar statistics only as an estimator of where errors come from, never as a decoding change.

Stage 2: test whether the budget is additive. Martin Hoferichter et al. (2023) found that individually sizable isospin-breaking effects largely cancel, leaving a small net effect. Word errors could behave the same way. A language model may already repair many phone confusions, so per-channel effects would not add up. We therefore test whether $\Delta_a$ predicts measured accented word error rate across accents and speakers.

Stage 3: allocate a recording budget. We use the budget to pick prompts. Selection is greedy: each step takes the prompt that most increases the coverage of channels, weighted by $w(c)\hat e_a(c)$, with diminishing returns per channel. This mirrors dense sampling near the $\rho$ peak.

Throughout, we use the finding of Prasad and Jyothi (2020) that accent information peaks in early layers and is smeared across phone boundaries for non-native speech. This argues for estimating channels from phone-level outputs aligned with a forced aligner rather than from intermediate features.

Baselines

Naive reference: uniform collection. For a fixed number of hours, record randomly drawn prompts from the target accent's pool. Every selector is compared to this at equal hours, using the same model, the same fine-tuning schedule and the same three random seeds.

Confidence-based active selection. Choose prompts or utterances where the current model's per-utterance confidence is lowest. This is the strongest generic competitor, because it needs no phonetic modeling and targets the model's weaknesses directly. It gets the same hours and the same tuning budget. No reported result was found on Lacuna for active data selection on accented speech, so we will establish this number ourselves.

Strongest modeling methods. The two strongest existing methods change the model rather than the data. The coupled training of Unni and Joshi (2023) penalizes distance between decoder context vectors of parallel utterances. It reports 53.21% word error rate on TEST-NONUS for the fine-tuned baseline and 28.39% for its best coupled system on Mozilla Common Voice. The Codebook Attend model of Prabhu et al. (2023) adds per-accent learned vector sets read through cross-attention. It reports up to 37% relative improvement on seen accents, up to 5% on unseen accents, and about 16% added inference time. Data selection is complementary to both. The fair test is therefore to train each of these methods on budget-selected data versus uniformly selected data at equal hours. Beating them means the selected-data variant lowers target-accent word error rate beyond the uniform-data variant by more than seed-to-seed spread.

Ablations of the key ingredient. The first ablation removes the control ratio and uses raw $\epsilon_a(c)$. The second removes the weight and sets $w(c)=1$. These isolate whether systematic cancellation and word-impact weighting each matter. Beating the budget's competitors means at least a 10% relative word-error reduction over uniform collection at the smallest budget, with confidence-based selection also beaten at that budget.

Why not just…?

Why not just use a large self-supervised model and skip data planning? CommonAccent (2023) shows fine-tuned XLSR reaching 97.1% accuracy on 16 English accents but only 68.5% on six closely related Spanish accents. Scale alone leaves fine-grained phonological differences unresolved, and the low-resource transfer difficulties reported by Rodolfo Zevallos et al. (2025) point the same way. We run every selector on top of an XLSR-based recognizer, so pretraining is held constant rather than set against the method.

Why not just use confidence-based active learning? This is the real threat. Low confidence may already track the high-$w\cdot e$ channels, and if so our method adds nothing except interpretability. The experiment settles it directly. If confidence selection matches budget selection at every budget size, we report that the error budget is useful for diagnosis but not for collection. One structural difference remains: confidence selection must first record and transcribe audio before it can score it. The budget selects prompts before anything is recorded, using only text and a small probe set. Our experiments therefore also compare the two at equal transcription cost, not just equal hours.

Why not just apply per-speaker normalization or parameter-efficient adaptation? Feature normalization removes channel and speaker effects but cannot say which sounds to record. Max Müller-Eberstein et al. (2024) note that parameter-efficient adaptation presumes knowing the target domain in advance, and choosing data well is exactly what provides that knowledge. The raw-error ablation tests whether the control ratio does anything beyond what normalization already gives.

Proposed Experiments

Experiment 1: estimator validity. On L2-ARCTIC we compute the rank correlation between $\hat e_a(c)$ and annotated substitution counts per native language. We do this with and without the control ratio, at probe sizes from 5 to 60 minutes per speaker. For reference, that corpus reports a phone-independent pronunciation-scoring precision-recall of 0.29, a low bar for phone-level error detection. The decisive check is whether the double ratio stays stable at small probe sizes, where the raw rate is dominated by speaker effects.

Experiment 2: additivity. On the Mozilla Common Voice accent subsets used by Prabhu et al. (2023) and Unni and Joshi (2023), we regress measured per-accent and per-speaker word error rate on $\Delta_a$. If the variance explained is small, the budget is not additive, as in the isospin-breaking cancellation, and Stage 3 is unjustified. That is a publishable negative result in itself.

Experiment 3: simulated collection. The transcribed Common Voice pool for each held-out accent serves as the "recordable" population. Each selector chooses 1, 2, 5 and 10 hours. We fine-tune a recognizer pretrained mostly on US English and score word error rate on disjoint speakers of that accent. The same selected subsets are then fed to coupled training and to Codebook Attend.

Experiment 4: zero-shot transfer. A model trained on one native language's budget-selected data is evaluated on the other L2-ARCTIC native languages. This checks whether channels shared across languages transfer, which matters given how weak gains on unseen accents have been (about 5% relative in the reported results).

Decisive checks. Two checks decide the outcome. First, the per-channel predicted gain must correlate with the realized word-error drop when we add data targeting only that channel. Second, budget selection must beat confidence selection at equal transcription cost. If both hold, physics-style error accounting gives limited accent coverage a principled recording plan. If either fails, we will know precisely which assumption broke.

References

  1. Measurements of the Continuum<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msub><mml:mi>R</mml:mi><mml:mrow><mml:mi>u</mml:mi><mml:mi>d</mml:mi><mml:mi>s</mml:mi></mml:mrow></mml:msub></mml:math>and<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mi>R</mml:mi></mml:math>Values in<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msup><mml:mi>e</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:msup><mml:mi>e</mml:mi><mml:mo>−</mml:mo></mml:msup></mml:math>Annihilation in the Energy Region between 3.650 and 3.872 GeV
  2. Precise Measurement of the<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msup><mml:mi>e</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:msup><mml:mi>e</mml:mi><mml:mo>−</mml:mo></mml:msup><mml:mo>→</mml:mo><mml:msup><mml:mi>π</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:msup><mml:mi>π</mml:mi><mml:mo>−</mml:mo></mml:msup><mml:mo stretchy="false">(</mml:mo><mml:mi>γ</mml:mi><mml:mo stretchy="false">)</mml:mo></mml:math>Cross Section with the Initial State Radiation Method atBABAR
  3. L2-ARCTIC: A Non-native English Speech Corpus
  4. Fully automated non-native speech recognition using confusion-based acoustic model integration
  5. Keyword Spotting Based on Phoneme Confusion Matrix
  6. Phenomenological Estimate of Isospin Breaking in Hadronic Vacuum Polarization
  7. How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems
  8. Coupled Training of Sequence-to-Sequence Models for Accented Speech Recognition
  9. Accented Speech Recognition With Accent-specific Codebooks
  10. CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice
  11. Assessing the Performance and Efficiency of Mamba ASR in Low-Resource Scenarios
  12. Hypernetworks for Personalizing ASR to Atypical Speech