Artifact-Free Replaced Token Detection for Heavily Fragmented Languages

Replaced token detection (RTD) pretrains a text encoder by having a small generator swap some tokens for plausible substitutes, then training the main model, the discriminator, to flag which tokens were swapped. Every position gives a training signal, which makes RTD far more compute-efficient than masked-word prediction. However, the swaps happen on subword pieces, and the generator fills each piece independently. In languages where one word is split into four or five pieces, a swap often produces a string that is not a word at all. We suspect that in these languages the discriminator learns mainly to spot malformed words, which is a spelling check, and not whether a real word fits its context, which is the semantic judgment that makes RTD useful.

This proposal does two things. First, it measures how much of the discriminator's detection ability actually passes through this "is this even a word?" cue, using a mediation analysis on counterfactual replacements. Second, it removes the cue with a simple change to how corruptions are made. The generator's output at a corrupted word is constrained so that the final surface word is one that already occurs in the pretraining corpus. The change needs no morphological analyzer and no new tokenizer. We expect it to matter most in the high-fertility languages that bear the "token tax". Token fertility here means the average number of subword pieces per word.

Core Claim

In a high-fertility language, a large share of the replacement-detection signal that RTD learns is carried by sub-lexical well-formedness. That share should grow with token fertility. Corrupting only into corpus-attested word forms cuts that path and yields better downstream encoders than standard RTD, whole-word-masked RTD, or a larger vocabulary under the same compute. In low-fertility English, the method should have little effect. The claim fails if the measured well-formedness share is small, or if the attested-word constraint does not beat whole-word masking plus a larger vocabulary at matched compute.

Why a missingness-graph view helps

The causal treatment of missing data, which models why a value is missing as part of the data-generating graph, gives a precise way to state the problem. Graphical Models for Inference with Missing Data represents each variable with three nodes: its true value $X$, a switch $R$ recording whether it was observed, and the recorded proxy $X^*$. It then asks which quantities can be recovered from what is actually recorded. RTD has the same shape. The original token is $X$, the replaced-or-not label is $R$, and the corrupted token the discriminator sees is $X^*$.

We add one node to this graph: $W$, whether the corrupted word is well formed. Under independent subword filling there is a path $R \to W \to X^*$. The discriminator can read $R$ off $W$ without using context at all. We want the discriminator to learn $R$ through the contextual fit of $X^*$, so the fix is to make $W$ independent of $R$ by construction.

The graph contains a second edge that RTD already has. Whether a masked position ends up labeled "replaced" depends on the true token itself, because predictable pieces are usually resampled correctly and therefore labeled "original". This is a self-masking edge $X \to R$, the case that Graphical Models for Inference with Missing Data and MIRACLE single out as hard. In fragmented words, the continuation pieces after the first piece are highly predictable. We therefore expect most of RTD's labeled replacements in these languages to fall at word-initial pieces and at the boundaries between pieces, which are exactly the places where malformed strings arise.

We borrow the lens, not the recoverability theorems. To Impute or not to Impute? shows in tabular treatment-effect estimation that how a missingness indicator relates to other variables decides whether it helps or corrupts inference. We apply the same reasoning to a corruption indicator in text. The research file notes that missingness graphs have not been adapted to token corruption processes in NLP.

Research Program

The measurement comes first. We take a trained ELECTRA-style discriminator (ELECTRA) and build counterfactual pairs at the same position in the same sentence. In each pair, the replacement word is either attested or unattested in the corpus, and the two are matched on generator probability so that contextual plausibility is held roughly fixed. The natural indirect effect of replacement on the discriminator's logit through $W$, divided by the total effect, gives the well-formedness share:

$$\text{WS} = \frac{E[\,D(X^*_{R=0,\,W=w_0}) - D(X^*_{R=1})\,] - E[\,D(X^*_{R=0,\,W=w_1}) - D(X^*_{R=1})\,]}{E[\,D(X^*_{R=0,\,W=w_0}) - D(X^*_{R=1})\,]}.$$

The symbols are as follows. $D$ is the discriminator's "replaced" logit. $X^*_{R=1}$ is the uncorrupted input. $X^*_{R=0,W=w_0}$ is a replacement that produces an unattested string. $X^*_{R=0,W=w_1}$ is a matched replacement that produces an attested word. The expectations run over positions, stratified by subword position within the word.

We then use this measurement to confront the "unexplained variance" pattern that recurs in the file. The head-ablation circuit in Decoded but Unused leaves 41% of a routing effect unaccounted for, and in The Token Tax token fertility explains at most half of the accuracy variance. Instead of asking a single scalar mediator to carry the effect, we fix the candidate pathway by construction and ask whether it carries the effect. We report the residual honestly.

The intervention is attested-word corruption. We build a word-type list from the pretraining corpus, using whitespace and punctuation splitting and a minimum frequency threshold. When a word is selected for corruption, the generator samples its pieces left to right, and the samples are restricted to prefixes of attested word types. This is the same as masking the generator's output with a prefix tree over the word list. Selection happens at the word level, with a per-word rate set so that the expected number of corrupted pieces matches standard RTD. Whole words can then be replaced by attested words that have a different number of pieces. That creates a new cue, the piece count, which we address by matching piece counts where possible and by testing in the mediation step whether the cue is actually used.

Baselines

The naive reference point is a standard ELECTRA-Small discriminator trained with subword-level random masking and independent generator filling, as in ELECTRA. It uses the same corpus, tokenizer, step count and hardware budget as every other arm, and it defines the gap we must close.

The strongest existing method in the file for strengthening RTD is TEAMS, from Training ELECTRA Augmented with Multi-word Selection. TEAMS adds a $(K+1)$-way task in which the discriminator must pick the original token from generator candidates. It reports GLUE 85.57 against 84.63 for ELECTRA-Base, and SQuAD 2.0 F1 of 84.51 against 81.52. TEAMS is the strongest competitor because it also attacks the "task too easy" failure, but through harder candidates rather than through well-formed ones.

We will reimplement TEAMS at Small scale on the same data. Each arm gets the same total pretraining FLOPs and a shared fine-tuning search over learning rate and epochs with the same number of trials. Beating TEAMS means a higher mean over at least five fine-tuning seeds on the Kinyarwanda tasks, with non-overlapping standard errors. We also test the combination, since the attested constraint can be applied to TEAMS candidates.

The morphology-aware reference is KinyaBERT. It replaces byte-pair encoding with an analyzer-driven two-tier encoder trained with masked prediction, and it reports gains of 4.3% on machine-translated GLUE and 2 to 3.2% on MasakhaNER over a byte-pair-encoding BERT. The file gives only relative gains, so we found no absolute score for this baseline on Lacuna.

KinyaBERT marks what a hand-built analyzer buys. We treat it as an upper reference, not a like-for-like competitor, because our method must work without an analyzer. Reaching its relative gain over the byte-pair-encoding masked-prediction baseline without an analyzer would count as a strong result.

Two ablations isolate the key ingredient. The first is whole-word selection with unconstrained independent filling. It changes where corruption happens but leaves the well-formedness path open. The second applies the attested constraint with subword-level selection. It separates the effect of the constraint from the effect of word-level selection.

Why not just…?

Why not just use whole-word masking for the generator? Whole-word masking decides which pieces are hidden. It does not stop the generator from filling those pieces independently into a non-word, so the $R \to W$ path stays open. It is a real threat if most artifacts come from partly corrupted words rather than from bad fills. The two ablations settle this: if whole-word selection alone matches the full method on downstream scores and drives the well-formedness share near zero, the constraint is unnecessary and we will say so.

Why not just retrain the tokenizer with a larger vocabulary? Lower fertility means fewer multi-piece words and so fewer chances to create artifacts. This is the most plausible competitor. The Roots of Performance Disparity argues that disparities shrink when segmentation is normalized. A larger vocabulary, however, spreads the same data over rarer embeddings, and in agglutinative languages it cannot absorb the combinatorial space of word forms. We include a byte-pair-encoding vocabulary two and four times the default, both with and without the constraint, at matched compute. The proposal's claim requires the constraint to help at every vocabulary size.

Why not just use a byte-level model or a morphological analyzer? The file documents the costs. Byte models pay a "byte premium" in sequence length, per The Roots of Performance Disparity, and need downsampling machinery like CANINE. Analyzer-based models like KinyaBERT exist only where an analyzer exists. Our method needs only a word list taken from the corpus, which exists for every language that has pretraining text.

Why not just add candidates as TEAMS does? Harder candidates raise difficulty, but if those candidates are themselves non-words, the discriminator can still pick the original by well-formedness alone. We measure the well-formedness share for TEAMS as well. If it is already low, our motivation weakens for that setting.

Proposed Experiments

We pretrain Small-scale discriminators on the Kinyarwanda corpus setting used by KinyaBERT and evaluate on its machine-translated GLUE, on MasakhaNER, and on the Kinyarwanda NEWS classification set. As a control we pretrain on English and evaluate on GLUE. We extend to additional MasakhaNER languages that span a range of fertility, so that the effect can be plotted against fertility.

A third probe comes from An Annotated Dataset of Errors in Premodern Greek, a morphologically rich language with annotated real errors. We score those errors zero-shot with the discriminator's replaced-token logit, and we fine-tune briefly against the dataset's detection baselines. Real errors are mostly well-formed misuses or near-miss forms, so a discriminator that relies on well-formedness should do poorly here.

The first decisive check is whether the well-formedness share is large for standard RTD in Kinyarwanda, rises with fertility across languages, and is small in English. The second is whether attested-word corruption drives that share down without lowering overall detection accuracy on attested replacements. The third is whether the downstream gains over standard RTD, whole-word RTD and TEAMS survive at the larger vocabulary sizes and scale with the drop in the share. We report whatever part of the downstream difference the share does not account for, rather than assuming the pathway explains everything.