Keeping Phonotactics Intact During Adversarial Domain Adaptation of Speech Recognizers

Speech recognizers break when the microphone, room, or speaker accent changes. A common remedy is domain-adversarial training. A second network, the discriminator, tries to tell whether an internal feature came from the original (source) recordings or the new (target) recordings. The recognizer is trained to fool it, so its features stop carrying information about recording conditions. The discriminator does not know which information is "recording condition" and which is language structure. Self-supervised speech models organize knowledge of phonotactics, the rules about which sound sequences a language permits (English allows /tr/ at the start of a word but not /tl/), in specific middle layers. An adversary attached at or above those layers may treat that knowledge as domain noise and erase it. The adapted model would then output phone strings that fool the discriminator but that no English speaker could pronounce.

We propose to test this failure directly and to fix it with two changes to standard adversarial adaptation. First, the discriminator is attached only below the layers where probing shows phonotactic structure appears. Second, on unlabeled target audio, the model's own output distribution is pushed to put its probability mass on phonotactically legal phone strings. The contribution is not "add a language model." It is a claim about representations: output-side fixes such as language-model fusion or constrained decoding cannot repair features that the adversary has already damaged. Protecting phonotactic structure during training should therefore close more of the domain gap than applying the same constraint only at decoding.

Core Claim

The research file supports three facts. Human-like Linguistic Biases in Neural Speech Models probes Wav2Vec2.0 with synthetic acoustic continua and finds that phonotactic preferences, such as resolving an ambiguous liquid toward /r/ after /t/, emerge around Transformer layers 4–5. The paper never studies domain shift. ALIGN shows that attaching an adversarial domain classifier to intermediate decoder layers is what makes cross-session speech decoding generalize. Its objective acts only on continuous features and never checks whether the decoded phoneme sequences are legal. Phonological Constraints and Morphological Preprocessing for Grapheme-to-Phoneme Conversion shows that enforcing simple syllable constraints in decoding (one vowel nucleus per syllable, one primary stress per word) cut German G2P word error from 21.5% to 13.7%. Phonotactic legality is therefore a strong structural signal that sequence models do not respect by themselves.

Our claim follows from these facts, and it is falsifiable. Adversarial alignment applied at or above the phonotactic layers measurably weakens the model's internal phonotactic bias and raises the rate of illegal phone strings on target data. A model that keeps the adversary below those layers and adds a target-side legality objective will reach lower target error than either the adversary alone or the adversary plus decoding-time phonotactic fusion.

Research Program

The base system is a Wav2Vec2.0-base encoder fine-tuned with CTC to emit English phones on the source domain of Libri-Adapt. CTC, or connectionist temporal classification, is a loss that lets a frame-level model emit label sequences without frame alignments. Libri-Adapt re-records LibriSpeech-clean-100 across six microphones, three accents (US, British, Indian English) and four noise conditions. It releases only DeepSpeech2 baselines and leaves adaptation algorithms open, which makes it a clean testbed. Phone targets come from dictionary pronunciations of the transcripts.

The work has three parts. The first is diagnostic. We train gradient-reversal adversaries, the standard minimax setup also used in Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images, attaching each one after a single layer $k$ from 1 to 12. For each run we measure three things: the continuum-probe phonotactic effect of de Heer Kloots and Zuidema at every layer, target phone error, and an illegal-string rate. The illegal-string rate is the fraction of decoded utterances that contain a syllable onset or coda unattested in a syllabified pronunciation lexicon. It is the speech analogue of the "syllable disintegration" documented for transliteration in A minimal-resource transliteration framework for vietnamese.

The second part is the method. We compile a weighted phonotactic automaton $A$ from lexicon syllabifications: a smoothed phone n-gram with zero weight on unattested onsets and codas and on nucleus-less syllables. The training objective is

$$\mathcal{L} = \mathcal{L}_{\text{CTC}}(S) - \lambda\,\mathcal{L}_{\text{dom}}(h_{\le k^*}) + \mu\,\mathcal{L}_{\text{phon}}(T), \qquad \mathcal{L}_{\text{phon}} = -\,\mathbb{E}_{x\sim T}\log \sum_{y} p_\theta(y\mid x)\,A(y).$$

Here $S$ and $T$ are the labeled source and unlabeled target sets. $\mathcal{L}_{\text{dom}}$ is the discriminator loss on encoder states $h$ at layers at or below $k^*$, the last layer before the probe shows phonotactic structure. $\lambda$ and $\mu$ are weights. $p_\theta(y\mid x)$ is the CTC probability of phone string $y$. The sum runs over all phone strings and is computed exactly by composing the CTC topology with $A$ and running the forward algorithm, the same way sequence-discriminative training builds a denominator graph. The phonotactic term needs no target transcripts, only a lexicon for the output language, and English phonotactics are shared across the accents being adapted to. The design follows the finding of Improved Reconstruction of Protolanguage Word Forms that pairing faithfulness with an explicit markedness (legality) term matters, and that ablating markedness hurts sharply.

The third part is a transfer check. ALIGN decodes English phonemes from neural recordings, and its outputs are subject to the same phonotactics. If the T12 data release permits, we add $\mathcal{L}_{\text{phon}}$ to ALIGN on T12 to test whether the effect holds beyond acoustic shift.

Baselines

The naive reference point is the source-only CTC model decoded greedily on each target domain. It establishes the size of the gap. For orientation, Libri-Adapt reports DeepSpeech2 word error rising from 10.19% (Shure, clean, US) to 24.95% under a shift to the Matrix Voice microphone, and from 11.4% to 29.5% under US-to-Indian accent shift. Those numbers come from a different architecture and serve only as a scale reference. We will rerun the source-only model on the same pairs.

The strongest existing adversarial method is ALIGN-style intermediate-layer gradient reversal, which we reimplement on Libri-Adapt. Its discriminator placement and $\lambda$ schedule are tuned on a held-out target-domain validation split with the same trial budget as our method. It is the strongest competitor because it is the only method in the file that shows intermediate-layer alignment beating strong sequence decoders on phoneme output. ALIGN reports session-average phoneme error of 30.46% against 38.74% for a Transformer baseline on T12, and held-out-session phoneme error of 43.39% against 52.61%. No reported result was found on Lacuna for any unsupervised adaptation method on Libri-Adapt, so our ALIGN-style run on that benchmark will be our own. On T12 the target is clear: beating 30.46% session-average phoneme error under ALIGN's own evaluation protocol.

The decoding-constraint baseline applies the same automaton $A$ at inference, by shallow fusion or hard pruning, on top of the adversarially adapted model. This follows the constrained-Viterbi approach of Schmid et al. and the inventory masking of Siminyu et al.. A phonotactically filtered pseudo-labeling baseline self-trains on target transcripts produced by the source model, keeping only hypotheses that are legal under $A$.

Two ablations isolate the key ingredients: the full method with the adversary moved above $k^*$, and the full method with $\mu=0$.

All systems share the encoder, the source data, the unlabeled target audio, the lexicon and the compute. Beating the field means lower target phone error than adversary-plus-fusion on a majority of Libri-Adapt shift pairs, with non-overlapping standard deviations over three seeds.

Why not just…?

Why not just adapt adversarially and add a phonotactic language model at decoding? This is the main threat. It uses exactly the same knowledge as $\mathcal{L}_{\text{phon}}$. The evidence against it as a complete answer is indirect but consistent. In Siminyu et al., constraining an unadapted model's outputs cut Saamia phone error only from 63.7% to 37.4%, while adapting the representation reached 9.2%. Output constraints recover little when the features are wrong. Our comparison of adversary-plus-fusion against our method-plus-fusion settles the question. If the gap vanishes once both use fusion, the representation-preservation claim is wrong and we will report that.

Why not just fine-tune on 100 labeled target utterances? Siminyu et al. found 100 utterances to be a sweet spot (Bukusu 41.2%, Saamia 15.5%). Labeled target speech is exactly what unsupervised adaptation assumes is missing, as it is for new BCI sessions in ALIGN. We still run 100-utterance fine-tuning as an upper reference to measure how much of the supervised gain the unsupervised method recovers.

Why not just self-train on pseudo-labels? Samir and Silfverberg show that selecting synthetic data by high loss preferentially picks phonologically illegal forms and costs 1.9% accuracy in Turkish. Uncertainty-driven target selection therefore has a known phonotactic failure mode. A legality filter could remove it, which makes filtered pseudo-labeling a serious competitor. It is included as a baseline, and if it matches our method, the simpler recipe should be preferred.

Why not just move the adversary to the bottom layers? That covers only half of the method, and it is precisely one of our ablations. The layer sweep will show whether placement alone restores phonotactic bias and error, or whether the legality term adds a separate gain.

Decisive Checks

The diagnostic succeeds if, across the layer sweep, adversaries attached at or above layers 4–5 lower the continuum-probe phonotactic effect and raise the illegal-string rate relative to lower placements, and if the illegal-string rate correlates with target phone error across runs. The method succeeds if it beats ALIGN-style adversary-plus-fusion on Libri-Adapt accent and microphone shifts while keeping the probe effect near the source-only model's level. The idea is refuted if decoding fusion alone closes the gap, or if the layer sweep shows no loss of phonotactic structure under adversarial training.