Speech recognizers do much worse on accents they rarely saw in training. A common fix is to make the model's internal representation "accent-invariant": a second network tries to guess the speaker's accent from the recognizer's hidden features, and the recognizer is trained to defeat it. The trouble is that accent labels are not the only thing that differs between accent groups in real corpora. The groups also differ in which vowels and words they happen to say, in where phrase boundaries fall, in speaking rate, and in speakers' baseline pitch and vocal-tract size. Many of these differences are linked by the physics of speech production. High vowels carry higher pitch, shorter duration and later pitch peaks than low vowels, and syllables stretch near phrase boundaries. An accent-guessing network can exploit all of these, so defeating it forces the recognizer to throw away the same cues it needs for ordinary, unaccented speech. We propose that this is one mechanical source of the familiar pattern in which accent robustness costs clean accuracy. The remedy is to remove only the accent information that these known confounds cannot explain.
Concretely, we condition the accent-guessing network on each frame's aligned phone, lexical stress, position relative to the phrase boundary, local speaking rate, and speaker-level physical attributes such as mean pitch and a vocal-tract length estimate. The recognizer is then penalized for carrying accent information beyond that context. This targets the conditional quantity $I(z; a \mid c)$ rather than $I(z; a)$, where $z$ is the hidden representation, $a$ the accent label and $c$ the confound context. The method needs no parallel recordings, so it works on spontaneous or rare-accent speech. The central empirical claim is testable. At matched accented word error rate, confound-conditioned invariance should cost measurably less clean accuracy than unconditional invariance, and it should keep native-speech sensitivity to stress and prosodic cues that unconditional invariance erodes.
What the literature already shows, and what it leaves open
The phonetics literature supplies the confounds and motivates the proposal. Matthias Jilka et al. measured pitch peaks in German at 35.16% of syllable duration for low vowels and 43.40% for high vowels, with mean durations of 139 ms versus 93 ms. Ho-hsien Pan et al. showed that unbalanced vowel height distorts spectral-tilt measures of voice quality. Petra Wagner et al. and Mitko Sabev et al. showed that unmapped phrase boundaries and focus inflate durations that would otherwise be attributed to something else. These are human-perception and production studies. None of them touches a trained recognizer. Our proposal carries their core control, balancing or conditioning on segmental and prosodic context, into the training objective of a speech model. Two further papers show that these cues carry word identity and so cannot be discarded freely: Mind the Peak, where an early pitch peak makes listeners hear museum as musical, and Sophia Fünfgeld et al., where accent placement shifts irony judgments.
On the modeling side, three invariance methods set the stage, and each leaves our question open. Sining Sun et al. apply domain adversarial training with a gradient reversal layer to Mandarin regional accents. The gradient reversal layer is a layer that flips the sign of the accent classifier's gradient so the encoder learns to confuse it. They report a 3.8% average relative character error reduction but never report standard-Mandarin accuracy. Aipnet pre-trains accent-invariant and accent-specific generators adversarially, and its gains are modest and hard to tune. Coupled training pulls together the representations of the same sentence read in two accents. Because both readings share the text, vowel content is automatically matched, which may explain why it avoids clean degradation. But it requires parallel recordings, which spontaneous speech and rare accents lack. Our method can be read as recovering coupled training's implicit control of phone content, and extending it to prosodic and speaker-physical confounds, without the parallel data.
The probing work constrains where to intervene. Archiki Prasad and Preethi Jyothi found accent information peaking in the first recurrent layer of DeepSpeech 2. LASAS found Conformer layers 4–9 best for accent recognition and showed that modeling accent as a deviation from text-derived anchors reduces channel shortcuts. That second idea, measuring accent relative to what the text predicts, is close in spirit to ours. LASAS, however, is a classifier, and it never asks what invariance does to recognition. The strongest accent-aware alternative, Codebook Attend, argues explicitly that adversarial invariance discards useful cues. We take that criticism seriously and ask whether it is a property of invariance itself or of unconditional invariance.
Finally, the robustness literature explains why shared features make the trade-off likely. Adversarial Neuron Pruning removes backdoors at a 1.47% clean accuracy cost on CIFAR-10. Yi Zeng et al. note that unlearning cannot separate weights serving legitimate and corrupting features. Sahar Karami et al. show that cleansing methods delete rare clean samples. Our hypothesis is the speech analogue of their observation. The "nuisance" (accent) and the legitimate signal (phonetic and prosodic cues) share channels, so an objective that cannot tell them apart removes both.
Core Claim
Unconditional accent-adversarial training removes accent-correlated but accent-irrelevant structure: phone composition, prosodic position, rate and speaker physiology. That removal accounts for a substantial share of its clean-speech cost. Conditioning the adversary on these variables should move the whole trade-off curve between clean and accented error, not just pick a better point on it. The confounds are defined once from forced alignments and pitch tracks, so the method adds no parallel data and no inference-time cost.
The objective is
$$ \mathcal{L} = \mathcal{L}_{\text{ASR}}(\theta) - \lambda\, \mathcal{L}_{\text{acc}}\big(D(z_\theta, c),\, a\big) + \mathcal{L}_{\text{acc}}\big(D_0(c),\, a\big), $$
where $\theta$ are the encoder parameters and $z_\theta$ their frame representations at a chosen layer. $c$ is the confound context. $D$ is an accent classifier that sees both $z_\theta$ and $c$, and $D_0$ is a classifier that sees only $c$. $\mathcal{L}_{\text{acc}}$ is cross-entropy against accent label $a$, and $\lambda$ sets the invariance strength. The encoder is rewarded only for making $D$ no better than $D_0$. That gap estimates accent information in $z$ beyond what the context already carries. Speaker identity is deliberately excluded from $c$. Speakers are nested within accents, so conditioning on identity would make the problem trivial. Using physical speaker attributes instead also targets the speaker–accent collapse reported in Synthetic Cross-accent Data Augmentation.
Research Program
The first step is a premise check that could end the project early. On L2-ARCTIC and Mozilla Common Voice accent subsets, we train $D_0$ alone and measure how well accent can be predicted from confound context without any acoustic representation. If $D_0$ is near chance, unconditional adversaries have little confound to exploit and the hypothesis is wrong. We also measure, on a frozen baseline encoder, how much of an unconditional accent probe's accuracy disappears when the probe is additionally given $c$.
The second step builds the method. We use a Conformer recognizer trained on LibriSpeech plus the Common Voice accent training split, matching the setting of Codebook Attend. We extract $c$ from forced alignment against the reference transcript during training, using canonical phone, vowel height class, lexical stress, a distance-to-pause proxy for phrase position, and local syllable rate, together with pitch tracking (speaker mean $f_0$) and a formant-based vocal-tract length estimate. Canonical rather than realized phones are used so that genuine accent substitutions, the thing we want removed from the recognizer's accent dependence, are not hidden inside $c$. The adversary attaches at the layers the probing studies identify. Following Qijie Shao et al., layers 4–9 are the main target, and an early block, following Prasad and Jyothi, is a secondary one.
The third step builds a clean-cue diagnostic, because word error rate alone may hide the cost we predict. It has three parts. The first is a native-speech stress and prosody probe: LibriSpeech words forming stress minimal pairs, plus phrase-final versus phrase-medial tokens, with linear probes on the frozen encoder for stress and boundary position. The second is the error rate on these minimal-pair words. The third is a dose-response test on the SIAEW corpus, whose five iso-perceptual steps of Spanish accent let us plot recognizer error against perceived accent strength. That separates accent strength from speaker and recording variation in a way Common Voice cannot.
Baselines
The naive reference is the same Conformer trained on pooled LibriSpeech and Common Voice with no accent-specific objective. It fixes the clean and accented operating point every method must improve on. A second naive reference is fine-tuning the pooled model on accented data. Melissa Torgbi et al. show that this lowers dialect error, from 33.6% to 24.0% WER on NESAC for Whisper large-v3, while introducing contextual substitutions. We therefore report both its accented gain and its clean and minimal-pair behavior.
The most direct competitor is unconditional domain adversarial training (Sining Sun et al.). It is run at the same layer, with the same adversary capacity (the conditional adversary's extra input is matched by widening the unconditional one) and the same schedule. Because a single $\lambda$ can be cherry-picked, both adversarial methods are swept over the same grid of $\lambda$ values with the same number of seeds, and compared as curves of clean WER against accented WER. Beating this baseline means a clean-WER reduction at matched accented WER that exceeds seed variance across most of the curve. Beating it at one point is not enough.
The strongest existing method is Codebook Attend (Darshan Deepak Prabhu et al.). On MCV_ACCENT it reports up to 37% relative WER improvement on seen accents and up to 5% on unseen accents, with no degradation on US English and about 16% extra inference time. We reimplement it on our splits with the same encoder size and training budget. Our claim against it is narrower. We aim to match its seen-accent gains within its reported range and exceed its unseen-accent gain at zero inference overhead. Failing to match on seen accents while winning on unseen ones would still be informative, and we would report it as such.
Where parallel text exists, in the shared Common Voice sentences and the ARCTIC prompts, we also run coupled training. It reports 28.39% TEST-NONUS WER against 53.21% for the best fine-tuning baseline. It serves as an upper reference for what full confound matching via parallel data can buy. Three ablations isolate the key ingredient. The first conditions only on canonical phone identity, testing whether segmental composition alone explains the effect. The second conditions on shuffled $c$, which keeps the parameter count while destroying the information, so any gain there is a capacity artifact. The third applies conditional invariance with prosodic context removed, isolating the phrase-boundary and rate variables motivated by the phonetics work.
Why not just…?
Why not just fine-tune on accented data? Fine-tuning is the cheapest route and often works on seen accents. The file shows its specific failure, though: contextual bias in Whisper (Torgbi et al.) and regressions when fine-tuning HuBERT (Synthetic Cross-accent Data Augmentation). Fine-tuning also does nothing specific for unseen accents. It is a baseline in our experiments, and the minimal-pair diagnostic will show whether its accented gains come with the cue loss we predict.
Why not just tune $\lambda$, or move the unconditional adversary to a less harmful layer? This is the real threat. If a carefully chosen $\lambda$ and layer for unconditional training reach the same clean/accented curve, conditioning adds nothing. We settle it by sweeping $\lambda$ and layer for both methods under identical budgets and comparing whole curves. The shuffled-$c$ ablation controls for the possibility that the conditional adversary simply acts as a weaker adversary.
Why not just normalize speakers at the front end, with per-speaker pitch, spectral and length normalization, and then use ordinary adversarial training? Normalization removes speaker-level physiology but cannot remove phone composition or phrase position, which differ by utterance. Blanket normalization of prosody would also strip cues that carry word identity (Mind the Peak). The broader filtering literature shows that input cleaning hurts rare-but-legitimate inputs (Avi Schwarzschild et al. report roughly 10% false rejection of benign prompts under perplexity filtering). We include normalization plus unconditional adversarial training as a baseline. If it closes most of the gap, the contribution shrinks to "physiology is the main confound," which is still a useful finding.
Why not just be accent-aware, as with codebooks, instead of invariant? Codebooks avoid erasure but rely on nearest seen accents, and their unseen-accent gain is small (5% relative). The two approaches are complementary rather than exclusive. As a follow-up we test conditional invariance underneath Codebook Attend, to see whether the unseen-accent gain adds.
Proposed Experiments
Training uses LibriSpeech plus Common Voice accent subsets. Evaluation covers LibriSpeech test-clean and US Common Voice as clean references, seen Common Voice accents, held-out Common Voice accents, L2-ARCTIC zero-shot, and Accented Vox Populi for unseen European accents. For Accented Vox Populi, no reported result found on Lacuna for any baseline WER, so all numbers there come from our own runs. The Scottish sets from Torgbi et al. are used if access permits. Metrics are WER per split, curves of clean against accented WER over the $\lambda$ sweep, stress and boundary probe accuracy on native speech, minimal-pair word error, SIAEW error as a function of iso-perceptual step, and conditional accent leakage, measured as the gap between $D$ and $D_0$ on held-out data.
Four checks are decisive, and each can falsify the proposal. The first is the premise test: accent must be predictable from $c$ well above chance. The second is mechanism: unconditional adversarial training must reduce native-speech stress and boundary probe accuracy relative to the pooled baseline, and conditional training must reduce it significantly less at matched accented WER. The third is outcome: the conditional method's curve must lie below the unconditional one on clean WER at matched accented WER across seeds, with the shuffled-$c$ ablation showing no such benefit. The fourth concerns scope. On SIAEW, error should rise more slowly with perceived accent strength for the conditional model. On unseen accents, the gain over the pooled baseline should exceed the 5% relative figure reported for Codebook Attend. If the mechanism check holds but the outcome check fails, the confounds exist but do not drive the trade-off, and we would report that as a negative result on a widely assumed explanation.
References
- The influence of vowel quality features on peak alignment
- Electroglottograph and acoustic cues for phonation contrasts in taiwan min falling tones
- Effects of Time Pressure and Spontaneity on Phonotactic Innovations in German Dialogues
- Bulgarian Unstressed Vowel Reduction: Received Views vs Corpus Findings
- Mind the Peak: When Museum is Temporarily Understood as Musical in Australian English
- Are You Being Sarcastic? Prosodic Cues to Irony Perception in German
- Domain Adversarial Training for Accented Speech Recognition
- Aipnet: Generative Adversarial Pre-Training of Accent-Invariant Networks for End-To-End Speech Recognition
- Coupled Training of Sequence-to-Sequence Models for Accented Speech Recognition
- How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems
- Linguistic-Acoustic Similarity Based Accent Shift for Accent Recognition
- Accented Speech Recognition With Accent-specific Codebooks
- Adversarial Neuron Pruning Purifies Backdoored Deep Models
- Adversarial Unlearning of Backdoors via Implicit Hypergradient
- Training Set Cleansing of Backdoor Poisoning by Self-Supervised Representation Learning
- Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition
- L2-ARCTIC: A Non-native English Speech Corpus
- Generating iso-accented stimuli for second language research: methodology and a dataset for Spanish-accented English
- Adapting Whisper for Regional Dialects: Enhancing Public Services for Vulnerable Populations in the United Kingdom
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models