Fitting the Noise, Not the Encoder: Adversarially Learned Character Corruption for Unlabeled Noisy Text

Language models break on misspelled, scanned, or dialectal text because their tokenizers split an unfamiliar spelling into unfamiliar pieces. The cheapest fix that works is to corrupt clean training text with random character edits (deletions, insertions, substitutions) so the model learns to ignore them. The edits are random, though, while real target text, such as a historical newspaper scanned with a Blackletter font or a Norwegian dialect, has its own specific error pattern. Usually we also have plenty of that target text, just without labels.

We propose using the standard tool for unlabeled target data, a domain discriminator (a small classifier trained to tell source text from target text), to tune the corruption process rather than the model. A tiny, interpretable character edit model is adjusted until a discriminator cannot tell corrupted clean text from real target text. Its output is then used as ordinary noise augmentation for an unchanged downstream training recipe. The discriminator's leftover accuracy also gives a diagnostic: it estimates how much of a target domain's shift is orthographic, and therefore how much any character-noise method can be expected to help.

Core Claim

In text, the adversarial game belongs on a few thousand parameters of a corruption channel, not on the hundreds of millions of parameters of a pretrained encoder. We make two predictions. First, a learned channel, mixed with random noise, will beat random noise alone at equal noise budget on targets whose shift is mostly orthographic, such as historical OCR and dialect spelling. It will do so without the instability of encoder-level alignment. Second, residual discriminability (how well a discriminator still separates learned-noise source from target) will predict, across target domains, how much character-noise augmentation can gain. Targets dominated by slang and lexical change, like social media, should show high residual discriminability and little gain.

The idea sits between two literatures that have not met in text.

The motivating evidence comes from noise-augmentation work. Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation used a fixed mix of edit operators and found by hand-ablation that deletion matters most. Cheap Character Noise for OCR-Robust Multilingual Embeddings showed that 5% random edits plus contrastive fine-tuning transfer to historical OCR better than targeted OCR simulators. Different Perturbations, Different Mechanisms showed that character noise in continued pretraining works by pulling dialect variants toward standard embeddings. All three fix the noise distribution in advance and never consult the unlabeled target corpus.

The method comes from adversarial domain adaptation. Unsupervised Domain Adaptation for Face Recognition in Unlabeled Videos used a three-way discriminator over clean images, synthetically degraded images, and real video frames. There the degradation was fixed and the feature network was adversarially aligned. We invert that arrangement: the corruption is learned and the encoder is never adversarially trained.

One paper constrains the design. The Michail et al. finding that random noise beats hand-built OCR simulators warns that a channel fit too tightly may generalize worse. We therefore regularize the learned channel toward random noise and treat the mixing weight as a central experimental variable rather than a nuisance.

Research Program

The corruption channel $q_\theta(\tilde x \mid x)$ maps a clean string $x$ to a corrupted string $\tilde x$. It has three parts:

  • per-character deletion probabilities;
  • an insertion distribution conditioned on the preceding character;
  • a sparse substitution matrix, initialized either uniform or keyboard-adjacent.

Altogether it has a few thousand parameters, all directly readable. A discriminator $D$, a small character-level convolutional network over windows of a few words, outputs the probability that a window comes from the target corpus. Because edits are discrete, the channel is trained by policy gradient, with the discriminator's log-odds as the reward:

$$\max_\theta \; \mathbb{E}_{x \sim S,\ \tilde x \sim q_\theta(\cdot \mid x)} \big[\log D(\tilde x)\big] \;-\; \beta\, \mathrm{KL}\big(q_\theta \,\|\, q_0\big)$$

Here $S$ is the clean source corpus, $q_0$ is the uniform random-edit channel of Michail et al., and $\beta$ controls how far the learned channel may drift from random noise.

One confound needs handling: the source and target corpora also differ in topic. The channel can only edit characters, so it cannot reduce topical differences. Those differences add a constant offset to the reward rather than a direction to follow. We still limit the discriminator's capacity and compute the reward relative to the unedited source window, so that a saturated discriminator does not drown out the gradient.

The downstream recipe is then left exactly as in the cited papers. We fine-tune the multilingual embedding models M-GTE and M-E5B with the Michail et al. contrastive loss on corrupted parallel text. For dialects, we run character-noise continued pretraining or noisy fine-tuning as in Srivastava & Chiang and MaiNLP. The only thing that changes is which corruption is applied. The augmentation is a mixture of learned and random noise with weight $\alpha$, and we sweep $\alpha$ from 0 (pure random) to 1 (pure learned).

The analysis component measures residual discriminability. We retrain a fresh discriminator after the channel converges and report its area under the ROC curve on held-out windows, for every target domain: HISTLUX, the CLSD Blackletter and salt-and-pepper subsets, the Bavarian, Neapolitan and Norwegian varieties in xSID, DialectBench clusters, and MTNT. We then test whether this number predicts per-target augmentation gain. MTNT serves as the deliberate negative control. Levy et al. found character noise does not help there because the shift is lexical, so our diagnostic should flag it as non-orthographic before any downstream model is trained.

Baselines

The naive reference point is the unadapted pretrained model, M-GTE or M-E5B for retrieval and mDeBERTa or the Llama-2/Qwen2 7B models for dialect tasks, evaluated directly on the target. It measures the size of the gap.

The strongest existing method on this question is random character noise as specified in the cited papers. For OCR retrieval this is 5% uniform edits with contrastive fine-tuning from Cheap Character Noise for OCR-Robust Multilingual Embeddings, which reports a Blackletter retrieval gain of over 4 points for M-GTE with no loss on clean semantic similarity. The file gives no exact absolute accuracy, and no reported number was found on Lacuna for HISTLUX. For dialects, it is character-noise continued pretraining from Different Perturbations, Different Mechanisms, which reports an average 3.0-point dialect gain, with character noise best in 6 of 9 settings. On Norwegian intent detection, MaiNLP at VarDial 2025 reports 97.6% accuracy for mDeBERTa with 20% character noise.

Fairness is enforced by holding constant the corrupted-token budget, the downstream training data, the training steps, the downstream hyperparameter search, and the random seeds (five per configuration). The random-noise baseline additionally gets its noise rate tuned over the same grid we use for $\alpha$, so that it receives the same tuning budget. Channel fitting cost is reported separately; we expect it to be well under one GPU-hour. Beating the baseline means a gain outside seed variance on HISTLUX and Blackletter retrieval and on dialect averages, with no clean-STS regression.

The Levy et al. four-operator mix is included as a second fixed-noise baseline. On the translation side, the reported reference is that it recovers German-English BLEU from 12.49 to 23.34 under natural noise.

Three ablations each isolate one ingredient:

  • Non-adversarial channel fit. The same channel is fit by matching character-bigram frequencies of the target corpus. This tests whether the discriminator adds anything beyond counting.
  • Encoder-level adversarial alignment. Following Sohn et al., we add a three-way discriminator with gradient reversal on encoder features over clean source, randomly corrupted source, and unlabeled target. This tests whether moving the adversary to the noise is better than putting it on features, and also measures training instability across seeds.
  • No regularization. We set $\beta=0$ to test whether overfitting the channel reproduces the Michail et al. warning about targeted simulators.

Why not just…?

Why not just use random character noise, which already works and costs twenty minutes? This is the most serious threat. Michail et al. found that random noise generalizes better than targeted OCR simulators, which suggests matching the target may not matter. Our design concedes this: random noise is both the regularization anchor and the $\alpha=0$ endpoint of the sweep. If the best $\alpha$ is zero on every target, the proposal fails on its first claim. Even then, the diagnostic claim stands as a separate, cheaply testable contribution. Our bet is that hand-built simulators failed because they were mis-specified, not because matching is harmful, and the $\beta$ ablation tests exactly that distinction.

Why not just run continued pretraining on the unlabeled target text, with or without noise? This uses the same unlabeled corpus more directly and is a genuine competitor for the dialect tasks. We run it as a baseline under the Srivastava & Chiang protocol, and we also run it stacked with learned-noise augmentation. If target-text pretraining alone matches our method, the corruption channel adds nothing for dialects. In that case the method's remaining value is for retrieval encoders and settings where pretraining a 7B model on target text is too expensive. The stacked run shows whether the two are additive.

Why not just align encoder features adversarially, as is standard? The research file notes that domain discriminators become unstable on high-capacity pretrained encoders. Indiscriminate alignment can also erase class structure when shift correlates with content, which is the case for dialect corpora with different topics. The Sohn-style ablation measures both the mean result and its variance across seeds. If it wins clearly, our claim about where the adversary belongs is wrong.

Why not just estimate the error channel from character statistics without a discriminator? Bigram matching is cheap and may capture most of an OCR confusion pattern. This is exactly the first ablation. If it ties the adversarial fit, we will report the simpler method as the recommendation.

Why not just swap in lower layers from a dialect expert? MaiNLP showed layer swapping reaching 85.6% slot F1 on Norwegian, beating character noise for slot filling. However, this needs a pretrained expert for the target variety. Historical Luxembourgish OCR and most DialectBench varieties have none, so we report it only where an expert exists, as an upper reference.

Proposed Experiments

The first decisive check uses planted channels with known ground truth. We corrupt clean German and French text with a hidden channel, such as a specific Blackletter confusion set (for example, long-s with f) or a dialect-like vowel rewrite. We use the result as an unlabeled target and test whether the learned channel recovers the planted parameters. We measure edit-probability error and whether the dominant learned operator matches Levy et al.'s finding that deletion matters most on natural typos. If recovery fails here, nothing downstream is interpretable.

The second check is the downstream comparison. We run the $\alpha$ sweep against all baselines on CLSD (minimal-noise, Blackletter, salt-and-pepper), HISTLUX, STS as the clean control, xSID intent accuracy and slot F1, WikiANN dialect NER, and SIB-200 Arabic topic identification. Success is a positive gain at the best $\alpha$ on orthographic targets with no clean regression.

The third check is the diagnostic. Across roughly fifteen targets, we correlate residual discriminability with the gain from any character-noise augmentation, using Spearman rank correlation. MTNT is the anchor. It has a reported clean-model English-to-French BLEU of 21.77 against 29.73 after in-domain fine-tuning MTNT, and Levy et al. saw no gain there from synthetic noise. MTNT should therefore show high residual discriminability and near-zero gain. A strong negative correlation would make residual discriminability a pre-training test for whether a team should bother with character noise at all.