When a speech recognizer moves to new speakers, accents, or noise, it does not fail evenly. It keeps getting "the" and "um" right and starts getting street names, drug names, numbers, and "not" wrong, which are exactly the words that decide whether a transcript is useful. The standard unsupervised fix for such shifts is adversarial domain adaptation. It uses no labels from the new setting: a small classifier, the "domain discriminator," tries to tell from the recognizer's internal features whether audio came from the old or the new setting, and the recognizer is trained to fool it. That process treats every audio frame alike. Most frames are silence, fillers, and function words, so most of the alignment effort goes where errors are cheap.
We propose to weight the adversarial alignment, frame by frame, by how semantically critical the underlying word is. Frames belonging to named entities, numbers, clinical terms, and negations receive most of the pressure to look domain-independent, and frames of low-stakes words receive little. We then test whether this reduces errors on critical words under real accent and noise shifts more than three things: uniform adversarial adaptation, generic task-gradient feature weighting, and simply fine-tuning on synthetic speech containing the critical words.
Core Claim
Two lines of work have so far been kept apart. The first is semantic error weighting, which scores recognition errors by their communicative cost instead of counting all edits equally. The second is adversarial domain adaptation, which learns features that do not reveal which domain the audio came from.
The first line shows that uniform counting misleads. Aggregate word error rate barely predicts whether humans can extract information from meeting transcripts (Automatic human utility evaluation of ASR systems). It agrees with clinician preferences only 10.5% of the time, against 75.9% for an entity-weighted BERTScore (Clinical BERTScore). Fixing misheard named entities and nouns recovers the most downstream QA and summarization accuracy, while fixing verbs recovers almost nothing (Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks). Almost all of this work is used only to evaluate models after training.
The second line shows that uniform alignment has a cost. Pushing whole feature distributions to match suppresses the fine-grained structure needed to tell classes apart (The Invariance Discriminability Tradeoff in Domain Adaptation; Lu et al.; Shi et al.). In speech, gains on the new domain came with losses on the original one (Adversarial Learning of Raw Speech Features).
Our claim is that these two findings are the same problem seen from two sides. If the alignment budget is spent on the rare, phonetically fragile, high-stakes frames, and the discriminator is not made to erase variation in frames whose errors cost nothing, then critical-token accuracy on the new domain should improve and damage to the source domain should shrink. The specific, testable prediction is that weighting by semantic criticality beats weighting by the recognizer's own gradients. If it does not, the semantic ingredient is unnecessary, and we will report that.
Relation to Prior Work
Four papers come closest. Each leaves a different piece open.
Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptation aligns clean-text and noisy-ASR-text embeddings inside a QA model. It never adapts the recognizer, and it weights all tokens uniformly. We adapt the acoustic model itself, which matters because a word the recognizer has already corrupted cannot be recovered downstream.
Adversarial Learning of Raw Speech Features and Unsupervised Speech Domain Adaptation Based on Disentangled Representation Learning adapt acoustic models to gender, accent, and noise shifts. The first uses a gradient reversal layer, the same trick used in Yu et al.: it flips the sign of gradients flowing back from the discriminator so the feature extractor learns to confuse it. The second separates content from style. Both treat every frame identically and report only word or phone error rate. They supply our base method, which we modify.
Backprop Induced Feature Weighting for Adversarial Domain Adaptation (BIWAA) is the closest in spirit. On image classification it weights features by classifier gradients so the discriminator aligns foreground objects rather than background. Our weights come from an external, human-grounded notion of error cost rather than from the model's own gradients. We treat BIWAA as the central competitor, not as prior art to set aside.
Evaluating and Improving Automatic Speech Recognition using Severity is the only training-time use of semantic weights in the file. It scales CTC loss by sentence-level embedding distance on single-domain LJ Speech. We move to token-level weights, apply them to the adaptation objective rather than to the supervised loss, and test under domain shift.
Two results constrain the design. First, the right notion of "critical" depends on the task: dialog-act classification relies on function words, not entities (Measuring the Effect of Transcription Noise). Our weight source is therefore a swappable component. Second, Zhang et al. and Nejjar et al. show that domain-specific variation sometimes carries task information. For that reason we down-weight alignment on low-stakes frames rather than removing all domain information everywhere.
Research Program
The recognizer is a pretrained CTC model, one that predicts a character or subword distribution for each audio frame. It has an encoder producing frame features $h_t$ and a frame-level domain discriminator $D$ attached through a gradient reversal layer. The training objective is
$$ \mathcal{L} = \mathcal{L}_{CTC}^{src} + \lambda \sum_t \tilde{w}_t \, \ell_{dom}\big(D(h_t), d_t\big), $$
where $\mathcal{L}_{CTC}^{src}$ is the supervised CTC loss on labeled source audio, $\ell_{dom}$ is the discriminator's cross-entropy on the domain label $d_t$, $\lambda$ sets the adversarial strength, and $\tilde{w}_t$ is the criticality weight of frame $t$. The weights are normalized to average one within each batch, so the total adversarial pressure matches uniform training and only its distribution changes.
Computing $\tilde{w}_t$ is the main engineering problem. On source audio we have transcripts. We tag each token as critical or not using an entity-and-number tagger, a negation lexicon, and, for medical data, a domain term list in the style of CBERTScore. We map tagged tokens to frames by CTC forced alignment.
On unlabeled target audio we must infer criticality from the model's own greedy hypothesis. This is circular: the entities most likely to be misrecognized are the ones least likely to be tagged. We handle this in three ways, and the ablation study compares them. The first variant tags the hypothesis and softens the weight by recognizer confidence. The second adds a lightweight frame classifier, trained on source alignments, that predicts "this region is critical" directly from acoustics, so it does not depend on the decoded text. The third uses an oracle that applies gold target transcripts and forced alignment. The oracle is used only for analysis, to bound how much is lost to imperfect weights.
A small floor weight keeps non-critical frames lightly aligned, and the floor is itself a tuned hyperparameter. As a secondary ingredient, source-side CTC loss can additionally be scaled per token by the same weights. We ablate this separately so that credit is not confused between the supervised loss and the adversarial loss.
Baselines
The naive reference point is the pretrained recognizer fine-tuned only on labeled source data and applied unchanged to the target domain. It shows how large the critical-token gap is to begin with. On the Streets data it is also anchored by the reported 44% street-name failure across 15 models ("Sorry, I Didn't Catch That").
The standard adversarial baseline is uniform frame-level adversarial training with a gradient reversal layer (Adversarial Learning of Raw Speech Features). It is reimplemented on our backbone with identical data, steps, and a $\lambda$ sweep of the same size. The published reference for this family is a Voxforge American-to-British word error rate of 23.10%, down from 28.11%. On CHiME4 the disentangled approach reports real-data development word error rate falling from 58.77% to 43.07% (Unsupervised Speech Domain Adaptation Based on Disentangled Representation Learning). We treat these as sanity targets for our reimplementation, not as numbers to beat directly, because the backbones differ.
The strongest adversarial competitor is BIWAA-style weighting ported to speech (Backprop Induced Feature Weighting). Here frame weights come from the magnitude of CTC-loss gradients at each frame, which is the generic "attend to what the task finds important" alternative. For vision it reports 71.5% Office-Home accuracy, and its weighting alone improved plain adversarial training by 9.7%. We give it the same normalization, floor, and tuning budget as our method. Beating it means lower target critical-token error at equal or better overall word error rate, with the difference holding across seeds.
The strongest non-adversarial competitor is supervised fine-tuning on fewer than 1,000 synthetic accented utterances that contain the critical words. This approach reportedly improved street-name accuracy by nearly 60% ("Sorry, I Didn't Catch That").
Finally, we include self-training on target pseudo-labels with severity-scaled CTC loss, following Evaluating and Improving Automatic Speech Recognition using Severity but at token level. This isolates whether the adversarial term contributes anything beyond semantic weighting of a supervised signal.
The key ablations are the three weight sources (confidence-softened hypothesis, acoustic criticality classifier, and oracle) and random weights matched in sparsity to the semantic ones. The random-weight ablation rules out the possibility that any sparse reweighting helps.
Why not just…?
Why not just fine-tune on synthetic speech containing the critical words? This is the real threat. The reported roughly 60% street-name gain from under 1,000 synthetic samples is large, and it may match or beat any unsupervised method when the list of critical words is known in advance. The two approaches cover different cases, though. Synthetic generation needs a closed list of target terms and a speech synthesizer that renders the target accent or noise faithfully. Adversarial adaptation needs neither. Our experiments therefore evaluate on held-out critical words absent from the synthetic set, and they also test the combination. If synthetic fine-tuning dominates even on unseen entities, we will narrow the claim to settings with no entity list.
Why not just give the recognizer the list of expected words? With a perfect context prompt listing all 29 street names, models still failed 24% of the time ("Sorry, I Didn't Catch That"). LLM correction is bounded by how phonetically close the hypothesis is to the truth (Aligning ASR Evaluation with Human and LLM Judgments). Both results place the failure in acoustic features under shift, which is where our method acts. We include prompted Whisper-Base as a reference, and also stack prompting on top of our model to test whether the two are complementary.
Why not just use BIWAA's gradient weights, which need no external tagger? Gradient magnitude marks frames that are hard to recognize, and those are not necessarily frames that are costly to get wrong. A disfluent filler can be hard to recognize and cost nothing. The speech-ported BIWAA baseline tests this directly. If it matches semantic weighting on critical-token error, the conclusion is that "task-attentive" alignment is enough and semantics adds nothing, and we will report it that way.
Why not just use a bigger model? Whisper-Large improves street-name accuracy over Whisper-Base, 73% versus 50%, but is 7 times slower and uses 10 times more memory ("Sorry, I Didn't Catch That"). Large speech foundation models also produce fluent negation flips, such as a word error rate of 0.16 alongside a semantic error score of 60.79 (Hallucination Benchmark for Speech Foundation Models). We report Whisper-Large as an upper reference on cost-unconstrained accuracy, not as a competitor at equal compute.
Proposed Experiments
The primary setting is accent shift toward non-native speakers on the SF Streets and US Streets datasets ("Sorry, I Didn't Catch That"), with native-speaker source speech. Target speakers are split into disjoint groups: one group is an unlabeled adaptation pool and the rest are the test set. Voxforge American-to-British and TIMIT male-to-female replicate the classic adversarial settings. CHiME4 real recordings test noise shift.
The downstream check uses two tasks. The first is Vietnamese medical spoken named-entity recognition on VietMed-NER, where F1 falls from 0.74 on clean text to 0.58 on transcripts with about 29% word error rate (Medical Spoken Named Entity Recognition). The second is QA over Spoken-SQuAD, where the text-level adversarial model reports 51.10 exact match and 63.11 F1 (Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering). On Spoken-SQuAD we ask whether adapting the recognizer beats, or adds to, adapting the QA model.
The headline metric is critical-token error rate: the error rate restricted to reference tokens tagged as entities, numbers, domain terms, or negations. For the Streets data we also report street-name accuracy and the resulting geographic misrouting distance. Every result is paired with overall word error rate, source-domain word error rate, CBERTScore, and the SHALLOW semantic-error axis. We do not score against the SHALLOW semantic axis during training, so that we do not simply optimize the evaluation metric.
Four checks decide the outcome. The first is whether criticality weighting lowers target critical-token error relative to both uniform and gradient-weighted adversarial training while keeping overall word error rate within noise. The second is whether it reduces the source-domain degradation reported for uniform alignment, which is the invariance-discriminability prediction. The third is how much of the oracle-weight gain the realistic weight sources recover. A large gap would mean the circularity problem dominates. The fourth is whether random sparse weights of the same density reproduce the gain. If they do, the benefit comes from sparsity rather than semantics, and the core claim fails.
A final task-dependence check swaps the weight source to function-word weighting and measures dialog-act accuracy on MRDA transcripts. The prediction is that the entity weighting that helps QA should hurt there, which would confirm that the weights carry task-specific meaning rather than acting as a generic regularizer.
References
- Automatic human utility evaluation of ASR systems: does WER really predict performance?
- Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings
- Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks
- Transferability vs. Discriminability: Batch Spectral Penalization for Adversarial Domain Adaptation
- When Unsupervised Domain Adaptation Meets Tensor Representations
- Information-Theoretical Learning of Discriminative Clusters for Unsupervised Domain Adaptation
- Adversarial Learning of Raw Speech Features for Domain Invariant Speech Recognition
- Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptation
- Unsupervised Speech Domain Adaptation Based on Disentangled Representation Learning for Robust Speech Recognition
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- Backprop Induced Feature Weighting for Adversarial Domain Adaptation with Iterative Label Distribution Alignment
- Evaluating and Improving Automatic Speech Recognition using Severity
- Adaptive Risk Minimization: Learning to Adapt to Domain Shift
- Uncertainty-guided alignment for unsupervised domain adaptation in regression
- "Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most
- Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approaches
- Hallucination Benchmark for Speech Foundation Models
- Medical Spoken Named Entity Recognition