Speech recognizers trained mostly on a few dominant accents make far more errors on everyone else. On the same sentences, a DeepSpeech 2 model scores 18.7% word error rate on Canadian speakers and 49.1% on Indian speakers (Prasad et al.). The usual fixes need accent labels: balanced sampling, accent-specific adapters, or worst-group optimization. Those labels are often missing or unreliable, as with the self-reported accents in crowd-sourced corpora. Focal loss is a way to spend more of the training signal on hard examples without knowing their group. It multiplies each example's loss by $(1-p)^\gamma$, where $p$ is the probability the model gives the correct answer. That shrinks the contribution of examples the model already gets right. Focal loss was designed for classification, though, and speech recognition is trained with sequence losses over whole utterances.
This note argues that the unit at which "hardness" is measured decides whether focal weighting helps accented speech. Accent errors are local: in L2-ARCTIC, annotators marked 5,199 phone substitutions among more than 349,000 phones, so a heavily accented utterance is mostly well-recognized sounds with a few substituted ones. We propose a label-free, frame-level focal variant of connectionist temporal classification (CTC, the standard alignment-free loss that sums over all ways to align the output characters to the audio frames). It re-weights each audio frame by how badly the model predicts the character that the soft alignment assigns to that frame. We test whether this narrows the gap between accents as much as group-level methods that require accent labels.
Core Claim
There are two tempting ways to add focal weighting to CTC, and both fail for different reasons. Applying it to the whole utterance, with $p = \exp(-\mathcal{L}_{\text{CTC}})$, fails because CTC loss grows with audio length. Long utterances get $p \approx 0$ and hence full weight regardless of accuracy. CTC-DRO showed this same length effect breaks group-level robust optimization. Applying focal weighting to raw frame posteriors without the alignment fails because frames predicted as "blank" dominate. The weighting then mostly redistributes gradient among blanks.
Our claim is that weighting at the frame level, measured against the alignment the CTC loss itself computes and normalized to a fixed average weight per batch, does two things. It is invariant to utterance length. It concentrates gradient on the few substituted or smeared phones that separate accented from native speech. As a result, it should reduce worst-accent error without accent labels, and it should get closer to accent-labeled robust optimization than utterance-level reweighting does.
The mechanism is simple to state. CTC's forward-backward pass gives, for each frame $t$ and output symbol $k$, an occupancy $\gamma_t(k)$: the probability that frame $t$ is aligned to $k$. The gradient of the CTC loss with respect to the frame logits equals that of the surrogate $-\sum_t \sum_k \mathrm{sg}[\gamma_t(k)] \log y_t(k)$, where $y_t(k)$ is the model's posterior and $\mathrm{sg}$ stops gradients. That identity lets us weight frames exactly:
$$ \mathcal{L}_{\text{fCTC}} = -\sum_t w_t \sum_k \mathrm{sg}[\gamma_t(k)] \log y_t(k), \qquad w_t = \frac{(1-q_t)^{\gamma}}{\overline{(1-q)^{\gamma}}}, \qquad q_t = \sum_k \gamma_t(k)\, y_t(k). $$
Here $q_t$ is the expected posterior of the aligned symbol, so it is low where the model disagrees with its own best alignment. The denominator is the mean unnormalized weight over all frames in the batch, so average weight stays at 1. This keeps focal weighting from acting as a hidden learning-rate decay as the model improves. It also keeps the result independent of how long the utterances are. With $\gamma = 0$ the loss reduces exactly to CTC.
Research Program
The work has three parts.
Build the loss and its variants. We implement frame-level focal CTC on top of a standard CTC fine-tuning recipe for a self-supervised encoder, using wav2vec 2.0 base and XLS-R 300M, the models studied by ACES and CTC-DRO. We also build a blank-aware variant. It uses a separate focusing strength for frames whose occupancy is mostly blank, borrowing the idea of splitting the loss by class from the decoupled cross-entropy of HMamba. HMamba applied that split, with a fixed weight, to phone-level mispronunciation detection rather than to sequence recognition.
Test whether the weights land where accent errors are. On L2-ARCTIC we force-align the canonical transcripts and record which frames receive high $w_t$. We then check whether those frames overlap the manually annotated substitutions, deletions, and additions more than chance does. We compare this overlap against the frames an utterance-level scheme would favor. This check tells us whether the method focuses on accent-driven errors or merely on noisy audio.
Map the over-suppression regime. Functional Properties of the Focal-Entropy proves two things that constrain the design. Focal loss reshapes the distribution it learns, pulling it away from the data distribution toward mid-range probabilities. And too large a $\gamma$ over-suppresses learning on very rare cases. We therefore sweep $\gamma$ and track two quantities. The first is the rarest accent's error, where over-suppression would show up. The second is per-accent calibration of token confidences, because the reshaped distribution could make the model's confidence scores less reliable for downstream use. AdaFocal, which tunes $\gamma$ per confidence bin using a validation set, reduced calibration error on CIFAR-100 from 5.64% with fixed focal loss to 1.36%. That result shows a single fixed $\gamma$ can miscalibrate. We treat AdaFocal-style adaptation as a possible follow-up only if the fixed $\gamma$ sweep shows calibration damage.
Where this sits relative to prior accent work matters. Accent-specific codebooks (Prabhu et al.), multi-basis adapters (Gong et al.), and accent-conditioned recognition (Deng et al.) all change the architecture and need accent identity during training. Coupled training (Unni et al.) forces representations of the same sentence in two accents to match. It needs parallel recordings, and its gains fell mostly on heavily accented samples even though it weighted every pair equally. That observation motivates our proposal.
Erasing accent directions from the representation is ruled out by ACES. There, partially projecting out the accent subspace raised clean word error rate by 0.3 points and widened the accent gap by 0.4 points, because accent and phonetic information are entangled. Our method leaves the representation alone and changes only where the gradient goes. It could in principle be combined with the architectural methods.
Baselines
Plain CTC fine-tuning. The naive reference point is standard CTC fine-tuning on the same imbalanced training mix, with no reweighting.
Accent-balanced sampling. The simple label-using baseline draws training utterances so that every accent contributes equally, using the accent labels the corpus provides.
CTC-DRO. The strongest competitor for the specific question of worst-group error is CTC-DRO, run with accents as the groups. It fixes CTC's length bias by comparing groups on fixed audio duration, and it smooths the updates to group weights. It reports up to 47.1% relative reduction in worst-language character error rate on ML-SUPERB 2.0. That result is multilingual; no reported result was found on Lacuna for CTC-DRO on accented English. CTC-DRO has an advantage our method lacks: it sees accent labels. Matching it without labels would therefore count as a strong result, and beating it would be a surprise we would scrutinize.
Architectural reference points. We cite Gong et al. (12% relative word error rate reduction on AESRC2020) and Prabhu et al. (up to 37% relative on seen and 5% on unseen accents on the Common Voice accent split, MCV_ACCENT). We do not re-run them as primary competitors, because they change the model. We do test one combination: codebooks plus frame-level focal CTC.
Ablations. Two ablations isolate the key ingredient. The first is utterance-level focal CTC computed on length-normalized loss, $p = \exp(-\mathcal{L}_{\text{CTC}}/T)$ with $T$ the number of frames. The second is frame-level focal weighting on raw, unaligned posteriors. Further ablations remove the per-batch weight normalization and remove the blank-aware split.
Fairness of comparison. All systems share the pretrained encoder, the training data, the number of updates, and a fixed tuning budget of the same number of trials. The trials cover $\gamma$ for ours and the step size and smoothing for CTC-DRO, selected on the same held-out split.
What counts as success. Beating the baselines means two things. First, lower worst-accent word error rate than plain CTC and utterance-level focal CTC, significant over three seeds. Second, worst-accent error within the seed spread of CTC-DRO, with average error no worse than plain CTC.
Why not just…?
Why not just use static inverse-frequency weights or balanced sampling by accent? This is a real threat whenever accent labels exist and are clean. Its weakness is that it treats every token of a rare accent as equally informative, and the L2-ARCTIC annotations say most of them are not. It also requires labels. The balanced-sampling baseline measures this directly. If it matches our method on AESRC2020, the honest conclusion is that frame-level focusing is worth using only where labels are absent or noisy. To test that condition, we include a variant with corrupted accent labels: we randomly reassign 30% of the accent labels in the training data.
Why not just use utterance-level focal loss on length-normalized CTC? This is the closest simple alternative, and it may work nearly as well. Per-frame average loss removes the length bias that CTC-DRO identified. Our argument against it is dilution. One substituted phone in a long, otherwise clean utterance barely moves the average loss, so the utterance gets little extra weight. The L2-ARCTIC overlap analysis and the head-to-head ablation settle this. If utterance-level focusing matches frame-level focusing on worst-accent error, we will report that the simpler method suffices.
Why not just run CTC-DRO over accent groups? When clean labels exist, it may well win on worst-group error, since it optimizes that quantity directly. It cannot target within-group difficulty, and it cannot run without group labels. Accents also blur into one another in ways fixed groups miss, such as a Canadian speaker with a Scottish parent. The unseen-accent evaluations (Spanish and Canadian in AESRC2020, and the nine unseen accents in MCV_ACCENT) test whether label-free focusing transfers better than weights learned per seen group.
Why not just remove accent information from the representation? ACES already found that projecting out the accent subspace hurts clean accuracy and fairness. Prabhu et al. found that adversarial accent-invariance training lost to accent codebooks on both seen and unseen accents. We therefore do not re-run erasure.
Proposed Experiments
The main testbed is AESRC2020, which has eight accents and 160 hours of speech. We make its training set deliberately imbalanced by keeping 100% of the American and British data and subsampling the Indian, Korean, and Portuguese data to 5%. The official test sets and the unseen Spanish and Canadian test accents stay unchanged. We also run the natural, balanced split. Relative results there matter because SAR-Net notes the corpus is roughly balanced across its accents.
The second testbed is the MCV_ACCENT split of Common Voice, with its five seen and nine unseen accents. Its crowd-sourced accent labels make it a natural test for the label-free setting. L2-ARCTIC serves only for the alignment analysis, because it has just 10 speakers.
We report five metrics: per-accent word error rate, worst-accent word error rate, the gap between the best and worst accent, average word error rate, and per-accent expected calibration error of token confidences.
Four checks decide the outcome. First, frame-level focal CTC must beat both utterance-level focal CTC and plain CTC on worst-accent error in the imbalanced setting, and it must reach CTC-DRO's worst-accent error without labels. If it does neither, the core claim fails. Second, on L2-ARCTIC, high-weight frames must overlap annotated mispronunciations clearly more than chance does and more than utterance-level weighting does. If gains appear without this overlap, the method is working through some other route, such as noise robustness, and we will say so.
Third, the $\gamma$ sweep should show a peak followed by degradation on the rarest accent. That pattern would confirm the over-suppression regime predicted by Functional Properties of the Focal-Entropy and set the practical range for $\gamma$. Fourth, the gains must survive the blank-aware ablation and the removal of weight normalization, and calibration must not get worse for the accents the method helps. If calibration does degrade, we test AdaFocal-style per-bin $\gamma$ as the repair.
References
- How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems
- L2-ARCTIC: A Non-native English Speech Corpus
- CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition
- ACES: Accent Subspaces for Coupling, Explanations, and Stress-Testing in Automatic Speech Recognition
- Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss
- Functional Properties of the Focal-Entropy
- AdaFocal: Calibration-aware Adaptive Focal Loss
- Accented Speech Recognition With Accent-specific Codebooks
- Layer-Wise Fast Adaptation for End-to-End Multi-Accent Speech Recognition
- Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-Supervised Learning
- Coupled Training of Sequence-to-Sequence Models for Accented Speech Recognition
- SAR-Net: A End-to-End Deep Speech Accent Recognition Network.