Aligning to What the Rules Miss: Residual Adversarial Adaptation for Dialects

Language models trained on a standard written variety, such as Modern Standard Arabic or Standard American English, perform much worse on regional dialects, and labeled dialect data is rarely available. Practitioners usually respond in one of two ways. The first is to rewrite standard training data into a dialect using hand-written linguistic rules. The second is to use domain-adversarial training: a small classifier (the "discriminator") tries to tell standard inputs from dialect inputs, and the main model is trained to fool it, which pushes the two kinds of input toward similar internal representations. Each approach fails in a known way. Rules capture grammar but miss spelling variation, slang, and social-media noise. Adversarial alignment between standard and dialect text tends to erase useful signal and can merge mismatched classes.

We propose to combine them so that each covers the other's weakness. Paired rule rewrites (a standard sentence and its dialectal transform) teach the model, through a simple consistency loss, to ignore the grammatical variation the rules describe. The adversarial discriminator is then aimed only at the residual gap, the difference between rule-synthesized dialect and real, unlabeled dialect text, rather than the whole standard-versus-dialect gap. The claim is that a discriminator should only have to remove what the rules could not supply. Pointing it at that narrower residual should give larger gains on natural dialect text, with less damage to standard-variety accuracy, than either augmentation alone or standard-versus-dialect adversarial alignment.

Core Claim

Standard adversarial adaptation asks the encoder to make standard and dialect inputs indistinguishable. Two well-documented problems follow. First, the discriminator has every dialect cue available to it, so the encoder is pressured to discard all of them, including cues that correlate with the label. Second, aligning marginal distributions merges mismatched classes. Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding documents this class-merging failure for plain domain-adversarial networks and repairs it with ALDA, a noise-correcting discriminator that does class-conditional alignment using pseudo-labels.

We take a different route. Most of the standard-to-dialect gap that rules describe can be closed without any adversary, because rule rewrites come in pairs with identical meaning and label. A consistency loss on those pairs is a direct, stable signal that tells the encoder exactly which variation to ignore. What remains is the gap between synthetic and natural dialect. That gap consists of orthography, lexical choice, and noise that no rule engine models; Multi-VALUE: A Framework for Cross-Dialectal English NLP acknowledges this limit for its 189 morphosyntactic rules. This residual is the only place where unpaired, distribution-level alignment is needed. It is also the place where the label-correlated "dialectness" signal is weakest, because synthetic and natural dialect text are both dialectal.

Concretely, with encoder $h$, task head $g$, standard labeled set $S$, rule transform $T$, and unlabeled natural dialect set $U$, the encoder and task head minimize

$$ \mathcal{L} = \mathcal{L}_{\text{task}}\big(S \cup T(S)\big) + \lambda_c\, \mathbb{E}_{x \in S}\,\lVert h(x) - h(T(x)) \rVert^2 + \lambda_a\, \mathbb{E}_{z \in h(T(S)) \cup h(U)} \mathrm{KL}\big(\mathcal{U} \,\Vert\, D(z)\big), $$

where the three terms are defined as follows. The first is ordinary cross-entropy on standard examples and their rule rewrites, which share labels. The second pulls each standard sentence's representation toward that of its own rewrite. The third pushes the discriminator $D$, which is separately trained with cross-entropy to separate synthetic-dialect from natural-dialect representations, toward the uniform distribution $\mathcal{U}$. This uniform target is the Random Prediction Training objective of Adversarial Adaptation of Synthetic or Stale Data, which showed that plain gradient reversal can overshoot and invert domain predictions instead of confusing them. Because the synthetic and natural pools can have different label proportions, the discriminator is also conditioned on the label: gold labels on the synthetic side and pseudo-labels on the natural side. This keeps alignment within classes, borrowing ALDA's noise-correction for the pseudo-labels.

Relation to Prior Work

The closest prior designs are adversarial systems that use an intermediate synthetic domain. Unsupervised Domain Adaptation for Face Recognition in Unlabeled Videos trains a three-way discriminator over clean images, synthetically degraded images, and real video frames. Adversarial Adaptation of Synthetic or Stale Data aligns developer-written synthetic utterances with live user queries using private and shared encoders. Both supply the method we reuse. Neither uses paired synthetic data to remove part of the gap before the adversary acts, and neither was applied to dialects.

The dialect-specific adversarial work gives the motivation and the strongest baseline:

Three pieces of non-adversarial dialect work constrain the design. Multi-VALUE reports that augmenting with rule-generated dialect text raises average cross-dialect F1 by 2.7 points but costs 1.2 points on standard English. DIALECTBENCH: An NLP Benchmark for Dialects, Varieties, and Closely-Related Languages finds that fine-tuning on small, noisy dialect data can widen the gap between standard and dialect performance. Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English shows that test-time entropy minimization with SHOT helps, but only on synthetic Multi-VALUE dialects. Together these mean the method must be judged on natural dialect text, with standard-variety accuracy tracked alongside.

Research Program

We build the method on two language pairs where both a rule transform and natural dialect data exist in the file.

For Arabic, the source is the MSA sentiment set ArSAS. The rule transform is the MSA-to-Egyptian morphosyntactic rule system of Emad Mohamed et al.. Natural Egyptian text comes from the unlabeled training portion of ASTD. The test target is ASTD, following ALDA's MSA-to-social-media protocol, with Moroccan (MSAC) and Tunisian (TSAC) as further targets. For the Maghrebi targets, the Egyptian rules act as a deliberately mismatched transform, which tests what happens when the rules cover little of the gap. The backbone is AraBERT, matching ALDA.

For English, the source is Standard American English CoQA. The transform is Multi-VALUE's Indian English and Chicano English rules. Evaluation uses Multi-VALUE's human-authored gold CoQA sets for those two dialects. The file lists no separate unlabeled natural corpus for these dialects. We therefore use a transductive split: half of the human gold inputs serve as unlabeled natural text, without labels, and the other half is held out for testing. Splits are rotated. The backbone is RoBERTa-base, matching Nguyen et al.

The analysis component asks whether the mechanism behaves as the claim predicts. Three diagnostics address this:

  • Residual size. Before adaptation, we train a frozen-encoder probe to separate synthetic from natural dialect. Its accuracy measures the residual gap. If rules cover the gap well, this accuracy should be near chance and the adversary should add little. If rules cover it poorly, as we expect for the Maghrebi targets, the adversary should matter more.
  • Feature erasure. We train linear probes for the task label on frozen representations from each method and test them on the standard variety. This shows whether standard-versus-dialect alignment erases more label information than residual alignment.
  • Class merging. We measure cross-class nearest-neighbor rates between natural-dialect representations and source class centroids, the failure ALDA was built to fix.

Baselines

The naive reference point is the source-only model: AraBERT fine-tuned on ArSAS, or RoBERTa fine-tuned on Standard American English CoQA, applied zero-shot to the dialect. Every method must beat it. Its gap to the target also gives the scale against which gains are read.

The strongest existing method on the Arabic question is ALDA (El Mekki et al.). It is the published adversarial approach for this exact source-to-target setting, and it already fixes class merging. The paper reports up to 91.5% accuracy on Egyptian sentiment and a 20.8% relative improvement over zero-shot AraBERT. We reproduce ALDA on our splits rather than relying on those figures, so that the comparison uses identical data. For fairness we also give ALDA access to the rule-augmented source set $S \cup T(S)$. Our method then differs only in what the discriminator separates and in the paired consistency term. Beating ALDA means higher target accuracy averaged over three seeds on ASTD, MSAC, and TSAC, with no larger drop on held-out ArSAS. On English, no adversarial baseline exists in the file. For SHOT on these dialects, only the synthetic-dialect result is available (an average adaptation gain of 0.98 on CoLA, RTE, and SST-2 in Nguyen et al.). We found no reported result on Lacuna for the human gold Indian English or Chicano English CoQA sets under any adaptation method, so all English numbers will be produced in-house.

The second strong competitor is rule augmentation combined with continued masked-language-model pretraining on the same unlabeled natural pool. It uses exactly the same ingredients as our method, without the adversary. Arabic Dialect Identification Using BERT-Based Domain Adaptation showed that this kind of pretraining helps: NADI development macro-F1 rose from 18.6% to 24.43%. The third competitor is SHOT test-time adaptation, run as in Nguyen et al. on the classification tasks.

All methods share the same backbone, the same labeled source data, the same unlabeled pool, and the same 24-trial tuning budget. Target-side tuning uses only a small labeled development slice, the same slice for every method.

Ablations isolate the key ingredient by changing one component at a time:

  • The discriminator target is varied among standard versus natural (the ALDA-style target), synthetic versus natural (ours), and the full three-way contrast of Sohn et al.
  • The paired consistency term is removed.
  • The uniform-target objective is replaced with gradient reversal.
  • Label conditioning is removed.

Why not just…?

Why not just augment with rule rewrites and continue pretraining on the unlabeled dialect text? This is the most serious threat, because it uses the same data with none of the adversarial instability. Masked-language-model pretraining adapts the encoder's vocabulary and token statistics. However, it never ties those adapted representations to the task head trained on standard data, and El Mekki et al. found that strong pretrained AraBERT features still leave a large cross-dialect gap that adversarial alignment closed. The experiment settles the question directly, because the combined baseline is run on identical data. If it matches our method on natural targets, the adversarial component is unnecessary and we will report that. A stacked variant (pretraining followed by our method) tests whether the two are complementary.

Why not just use test-time adaptation like SHOT or EMEA? These methods need no training-time changes. Their limits are documented, though. SHOT applies only to single-label classification, so it cannot handle span-extraction CoQA. EMEA's test-time gradient steps cut throughput from 250 to 9 examples per second (Efficient Test Time Adapter Ensembling for Low-resource Language Varieties). The positive SHOT evidence also comes only from synthetic dialects. We run SHOT on the Arabic sentiment targets, where it applies. If it matches us there, our advantage reduces to task coverage and inference cost, and we will say so.

Why not just use ALDA, or the three-way discriminator, with augmented data added? These are our ablations. The prediction that separates the methods is specific. Residual alignment should beat standard-versus-natural alignment by a larger margin when the rules cover more of the gap, as with Egyptian rules on Egyptian text. That advantage should shrink on the Maghrebi targets, where the residual is nearly the whole gap. If residual alignment never beats the three-way contrast, then the narrowing idea adds nothing over simply adding a synthetic domain.

Proposed Experiments

The decisive comparison is target accuracy on natural ASTD, MSAC, and TSAC, and F1 on the human gold Indian and Chicano English CoQA sets. Each method is reported alongside its accuracy change on the standard variety, so that gains bought by sacrificing standard performance are visible. Multi-VALUE's 1.2-point loss under augmentation is the reference for an acceptable cost.

Two checks could falsify the core claim. First, if feature-erasure probes show no difference in retained standard-variety label information between standard-versus-natural and synthetic-versus-natural discriminators, the stated mechanism is wrong even if accuracy improves. Second, across the five target dialects, the gain from the adversarial term should rise with the residual-size probe accuracy measured before adaptation. A flat or negative relationship would mean the adversary is not doing what we claim. We also report seed variance and the discriminator's accuracy trajectory during training, since instability is the standard objection to adversarial methods. The method should hold up without per-target retuning of $\lambda_a$.