Many speech systems feed a model two views of the same words: the spelled text and a phoneme sequence, meaning a string of sound units produced by an automatic pronunciation converter called a grapheme-to-phoneme (G2P) tool. The phoneme view helps with words that sound alike but are spelled differently. The converter also makes mistakes, and those mistakes flow into everything downstream. The usual fix is to train on noisy phonemes until the output stops getting worse. We argue that good output can hide two very different internal behaviors. In the first, the model learns selective tolerance: it ignores the particular phoneme tokens that are wrong and keeps using the correct ones. In the second, it learns global suppression: it quietly stops trusting the phoneme stream at all, which gives up the very benefit phonemes were added for. Immunology draws the same line between targeted tolerance and systemic immunosuppression, and the distinction is the main thing we borrow from it.
The proposal makes the distinction measurable. We inject controlled, realistic G2P errors into a phoneme-augmented ASR error corrector. We train probes, which are small classifiers fit on frozen hidden states, to find where the model internally registers that a phoneme is wrong. We then use causal erasure to measure how much the model relies on the phoneme stream at clean tokens versus corrupted ones. That ratio tells tolerance apart from suppression. Only after the diagnostic is in place do we ask whether routing the probe's signal back into the model as a token-level gate produces tolerance more reliably than the simpler options.
Core Claim
Output metrics such as word error rate reduction cannot tell selective tolerance from global suppression. A token-level causal reliance measure can. We expect common robustness recipes, especially training with noisy phonemes, to push models toward suppression.
We define reliance $R$ on a set of tokens as the drop in correct-edit accuracy when the phoneme-encoder representation at those tokens is replaced by its mean, using the mean-projection or LEACE erasure in Improving Causal Interventions in Amnesic Probing with Mean Projection or LEACE. We then define the selectivity index $S = R_{clean} - R_{corrupt}$, where $R_{clean}$ and $R_{corrupt}$ are measured on tokens whose phonemes are correct and incorrect respectively. The three regimes read off directly:
- A model that absorbs the noise has both reliances high and $S$ near zero.
- A suppressed model has both reliances low.
- A tolerant model has high $R_{clean}$ and $R_{corrupt}$ near zero.
The immunology file motivates why abundance is the wrong thing to count. Regulatory T cells can be present at normal frequency yet fail to suppress (Analysis of frequency changes...; Tuning the Cellular Rheostat for Immune Tolerance). Low-dose IL-2 works by acting selectively on high-affinity receptors on regulatory cells rather than dampening the whole system (Case report: Low-dose interleukin-2...). The machine-learning analogues are that "the phoneme stream is present in the input" is not the same as "the phoneme stream is functionally used," and that a good intervention should be token-selective rather than global. These papers supply the question. They do not supply a mechanism, and nothing here depends on the biology being literally analogous.
Research Program
Testbed. We use the setting of PATCorrect. PATCorrect is a non-autoregressive ASR error corrector with two parallel encoders, one for text and one for phonemes derived by G2P from the ASR hypothesis. The two are fused by cross-attention, and a tag predictor chooses keep, delete, substitute, or add for each token. PATCorrect showed that phoneme fusion helps, but it treated the G2P output as given and never examined how converter errors are handled inside the model.
We train on the $S^{trans}$ corpus of ASR hypotheses built from LibriSpeech and Common Voice. We regenerate the phoneme stream under three conditions:
- dictionary pronunciations where available, which we treat as clean;
- the default converter;
- controlled corruption.
For controlled corruption, we substitute phonemes at a set rate, sampling replacements in proportion to articulatory similarity under PanPhon features. The articulatory-feature probing work of Probing Subphonemes in Morphology Models supplies this feature machinery. Similarity-weighted substitution makes the errors resemble real converter confusions rather than uniform noise.
Diagnostic stage. First, we fit per-layer linear probes on the phoneme encoder, the fusion layers, and the decoder to predict "this token's phoneme is corrupted." We report minimum-description-length (MDL) code length alongside accuracy, as Astrach and Pinter do. A probe that only memorizes would show a long code length, so MDL guards against that.
Second, we compute $R_{clean}$, $R_{corrupt}$, and $S$ at each layer.
Third, we run activation patching, swapping in clean-condition phoneme states for corrupted ones at one layer at a time. This follows the causal localization in Beyond Transcription, which traced most Whisper repetition loops to a single attention head and detected hallucinations with a linear probe at a late decoder position. We extend that style of analysis from one end-of-sequence state to per-token error propagation through a fusion interface.
We deliberately start with linear probes. The input here is discrete phoneme tokens rather than raw spectrograms, so the failure of linear probes on audio self-supervised models reported in BAT may not carry over. If it does carry over, we switch to its convex gated probe.
Intervention stage. If a layer exists where corruption is linearly decodable before the fusion step, we freeze that probe. Its per-token output then scales the cross-attention weights going to the phoneme stream. This is a token-level dial, the opposite of a global switch. We also run a second, smaller study on the PATCorrect decoder. That decoder queries text first and then phonemes, and we test whether reversing this order changes $S$. The question comes from the order dependence reported in TIGIT reverses IFN-α-promoted Th1-like Tregs. We treat it strictly as a hypothesis generator.
Cross-lingual replication. We repeat the diagnostic on the SIGMORPHON 2017 inflection models from Astrach and Pinter, injecting G2P errors into their IPA input. The question is whether articulatory features stay decodable in context layers when a single input phoneme is wrong. For example: does Hungarian vowel-harmony information get repaired by context or corrupted?
Baselines
Text-only reference. The naive reference is a text-only non-autoregressive corrector with the same architecture minus the phoneme encoder. It sets the floor that phonemes must beat. A model that has fully suppressed its phoneme stream should converge toward this floor, which makes the comparison diagnostic in its own right.
PATCorrect as published. The strongest existing method is PATCorrect, which reports an 11.62% word error rate reduction (WERR) on its test set, against 9.46% for FastCorrect (PATCorrect). We retrain it on our regenerated phoneme streams rather than comparing to that number directly, because our corruption conditions change the input. It is the strongest competitor because it already shows cross-attention fusion beats simpler fusion.
PATCorrect with noise augmentation. We train PATCorrect on corrupted phonemes at matched rates. This is the recipe a practitioner would actually try, and it is the one we predict will raise WERR while lowering $R_{clean}$, which would be suppression.
Learned gate. A per-token sigmoid gate on the phoneme cross-attention is trained end to end with no probe. This is the key ablation: it asks whether the probe adds anything beyond giving the model a gate.
Shuffled-probe gate. The probe-gated model is run with its probe outputs shuffled across tokens. This controls for the extra parameters and the act of gating itself.
Fairness. All systems share the training data, step count, model size, and a fixed budget of hyperparameter trials.
Criteria for "beating" the baselines. Beating PATCorrect-with-augmentation means three things at once:
- equal or better WERR on corrupted test streams;
- WERR on clean streams no worse than PATCorrect trained clean;
- a significantly higher $S$ across three seeds.
Why not just…?
Why not just train with noisy phonemes? It is cheap, and it may give the best WERR. Our claim is that it can buy WERR through suppression. An Illusion of Unlearning? shows that output behavior after fine-tuning can misreport what is happening inside a model, which is exactly why WERR alone cannot settle this. The comparison of $R_{clean}$ between the clean-trained and augmentation-trained models settles it. If augmentation preserves high $R_{clean}$ and drives $R_{corrupt}$ to zero, then augmentation already induces tolerance and our intervention is unnecessary. We would report that outcome as a finding.
Why not just let a learned gate figure it out? This is the real threat. An end-to-end gate may learn the same signal the probe finds. The learned-gate baseline exists to test this. If it matches the probe-gated model on both $S$ and WERR, we drop the intervention claim and keep the diagnostic contribution: the selectivity measure and the localization of where errors are registered. The probe-gated design has one testable advantage. Its corruption detector is fixed and auditable, so we can check whether a learned gate's values actually correlate with true corruption labels.
Why not just use the converter's confidence or a pronouncing dictionary? Grapheme-Coherent Phonemic and Prosodic Annotation shows how static, dictionary-side constraints can fail. Dictionary pruning raised G2P match rate on JSUT-TTSaug from 38.55% to 81.47%, but phoneme error rate rose from 0.21% to 1.12% when the upstream ASR text was wrong. More basically, our phonemes are derived from ASR hypotheses that may themselves be misspelled, so converter confidence on a wrong word can be high. We still include a confidence-gated variant at no extra cost.
Why not just drop phonemes? Evidence in the file argues against it. MMSpeech reports a 48.3% relative WER reduction over HuBERT on Mandarin from an explicit phoneme bridge. PnG BERT reports that the combined phoneme-plus-text input reached a MOS of 4.47, matching ground-truth recordings, against 4.41 for its baseline. The text-only reference baseline measures what dropping phonemes costs in our own setting.
Proposed Experiments
The first decisive check is whether $S$ separates models that WERR cannot. We would show this if the clean-trained and augmentation-trained PATCorrect variants land within one point of WERR on corrupted streams yet differ substantially in $R_{clean}$. If $S$ never separates models that WERR already separates, the measure adds nothing and the project stops there.
The second check is localization. Corruption should be decodable by a linear probe with short MDL code length in the phoneme encoder before fusion, and activation patching at that layer should recover a large share of the WERR lost to corruption. If corruption becomes decodable only after fusion, a pre-fusion gate is impossible, and the intervention stage is abandoned.
The third check is the head-to-head among the gates (probe-gated, learned, confidence-based, and shuffled) at corruption rates swept from zero to high. We report WERR on clean and corrupted streams, $S$, and the correlation between gate values and true corruption labels.
The fourth check is the order-of-querying study, measuring $S$ under text-first and phoneme-first decoding.
The fifth check is the SIGMORPHON replication. It asks whether context layers repair or propagate a single corrupted phoneme's articulatory features across seven languages, reported as MDL compression relative to the clean-input probes of Astrach and Pinter. No reported per-language compression numbers were found on Lacuna for that paper, so we will recompute its clean baseline ourselves.
References
- Improving Causal Interventions in Amnesic Probing with Mean Projection or LEACE
- Analysis of frequency changes in CD8+ regulatory T cell subsets in peripheral blood of individuals with type 1 diabetes
- Case report: Low-dose interleukin-2: a treatment of bullous pemphigoid with predominantly perifollicular blistering caused by PD-1/PD-L1 inhibitor
- PATCorrect: Non-autoregressive Phoneme-augmented Transformer for ASR Error Correction
- Probing Subphonemes in Morphology Models
- Beyond Transcription: Mechanistic Interpretability in ASR
- BAT: Better Audio Transformer Guided by Convex Gated Probing
- TIGIT reverses IFN-α-promoted Th1-like Tregs via in-sequence effects dependent on STAT4
- An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations
- Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning
- MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for speech recognition
- PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS