Subword tokenizers split words into pieces by frequency, not meaning. This makes models brittle. A typo, an unusual inflection, or a spelling variant can change how a word is split, and the model then sees a token sequence it rarely saw during training. Adversarial training, which means training on the inputs that hurt the model most, is the standard fix for brittleness. In text it is expensive, because finding the worst input requires searching a huge discrete space on every training step. We propose restricting that search to the few ways a word can be split along its morpheme boundaries. A morpheme is the smallest meaningful unit, such as "un-" or "-ness". A morphological segmenter proposes these splits, and the model picks the most damaging one. The segmenter is used only during training. At test time the model runs with its ordinary tokenizer, so errors from the segmenter on noisy input never reach deployment.
The bet is that morphologically plausible re-splits are a small search space, often two to four candidates per word. They are also the space that realistic noise lands in, because inflection changes and many typos move exactly those boundaries. If so, worst-case training over this small set should match most of the robustness gain of unconstrained adversarial re-splitting at a fraction of the cost. It should also clearly beat random re-splitting, and the advantage should be largest in languages where words fragment into many pieces.
Core Claim
Let $x$ be a training input, $y$ its target, and $\mathcal{S}_{\text{morph}}(x)$ the set of token sequences obtained by re-splitting a few selected words of $x$ only at boundaries proposed by an unsupervised morphological segmenter, with each morpheme tokenized by the model's own subword vocabulary. We train on
$$\min_\theta \; \mathbb{E}_{(x,y)} \Big[ \mathcal{L}(f_\theta(x),y) + \lambda \max_{s \in \mathcal{S}_{\text{morph}}(x)} \mathcal{L}(f_\theta(s),y) \Big],$$
where $f_\theta$ is the model, $\mathcal{L}$ the task loss, and $\lambda$ weights the adversarial term.
We make three claims, each testable. First, at matched training compute, this beats random subword regularization in robustness to character noise and real user noise. Second, it roughly matches adversarial selection over an unconstrained statistical candidate pool while using several times fewer forward passes. Third, the gains grow with fertility, the average number of subword pieces per word. If unconstrained adversarial splits turn out to be clearly stronger, the morphology constraint is costing robustness rather than buying efficiency, and the claim fails.
Relation to Prior Work
The closest work is Adversarial Subword Regularization for Robust Neural Machine Translation (ADVSR). ADVSR samples candidate segmentations from a SentencePiece unigram model, scores them with gradients and loss evaluations, and trains on the worst one. It reports +3.2 BLEU on IWSLT Czech→English and up to +2.3 BLEU on the noisy MTNT set. ADVSR supplies our method template. We change two things: the candidate pool, which becomes morphological rather than statistical, and the search budget. We also ask a question ADVSR does not: whether the constraint trades robustness for efficiency.
Tokenization Falling Short motivates the work. It documents an "embedding disconnect", a cosine similarity of 0.21 between "assignment" and "assign"+"ment", and shows that fine-tuning with random BPE-dropout helps. Its dropout is random and unconstrained. Our method is targeted and constrained.
Adversarial Tokenization shows that models understand non-canonical splits of the same string. It also shows that safety training ignores such splits, and it notes that training over all of them is prohibitive. We treat this as motivation, but deliberately do not target its same-string threat, for reasons given below.
On the hybrid tokenization side, Exploring morphology-aware tokenization trains BPE on MorphAGram-segmented Spanish. It improves SyntaxGym ES agreement from 74.5% to 92.6% and LAMBADA ES from 33.8% to 40.0%. MVP-BERT shows that multi-granularity vocabularies raise resistance to TextFooler. Both bake linguistic segmentation into the deployed tokenizer. They therefore inherit the cascading-error problem, in which the upstream segmenter misparses noisy text. Our proposal uses morphology only as a training-time source of perturbations.
Finally, Towards Improving Adversarial Training of NLP Models (A2T) makes word-substitution adversarial training tractable. It ranks words with one gradient pass, uses cheaper constraint models, and attacks a 20% subset per epoch, giving nearly a 3x speedup over TextFooler-based training. We reuse its gradient ranking and subset scheduling for a different perturbation type.
Research Program
We will build a training-time perturbation generator with three parts. The first is an unsupervised morphological segmenter, MorphAGram, trained per language on the same corpora used for the model. It produces morpheme boundary hypotheses for each word type, cached once.
The second part scores words by saliency. Following A2T, one forward-backward pass computes the norm of the loss gradient for each word's subword embeddings. The top-$k$ words, with $k$ around 10–15% of words, are chosen for re-splitting.
The third part selects the worst candidate. For each chosen word, we enumerate its morpheme-boundary candidates, capped at $K \le 4$ and tokenized within morphemes by the existing vocabulary. We evaluate them in a single batched forward pass and keep the worst combination under a greedy left-to-right pass.
Adversaries are regenerated on a subset of data each epoch. Cost is reported as forward-pass equivalents per training token and as wall-clock time on fixed hardware.
We study three levers. The candidate source is morphological, statistical (the SentencePiece pool, as in ADVSR), or a union of the two. Selection is adversarial or random within the same pool. The budget in $k$ and $K$ is swept to trace a robustness-versus-compute curve. The main object of analysis is that curve, per language, plotted against each language's measured fertility.
Baselines
The naive reference is a standard BPE model trained on clean data with the same architecture, data, and steps. It anchors both clean accuracy and noisy-input degradation.
The first real competitor is random subword regularization: BPE-dropout at $p=0.2$ as used in Tokenization Falling Short, plus SentencePiece sampling. It is the strongest cheap alternative because it costs almost nothing per step. We give it equal wall-clock time, meaning more epochs, and tune its dropout rate on the same development noise as our $\lambda$. Beating it means higher BLEU under the noise sweep and on MTNT at equal compute, not just at equal steps.
The strongest existing method is ADVSR Adversarial Subword Regularization for Robust Neural Machine Translation. We reimplement it on the same Transformer-base translation setup and data. The reported numbers we measure against are +3.2 BLEU over baseline BPE on IWSLT Czech→English and up to +2.3 BLEU on MTNT English→French. The file gives these as gains, not absolute BLEU, so our reimplementation must first reproduce comparable gains over our own BPE baseline. "Beating" ADVSR means matching its noisy-input BLEU within about 0.5 while using at most half its forward-pass budget, or exceeding it at equal budget.
A static hybrid tokenizer serves as the ablation for "morphology helps anyway". This is the MorphAGram-then-BPE tokenizer of Exploring morphology-aware tokenization, used at train and test time without adversarial training. The ablation that isolates our key ingredient, adversarial choice, is random selection within the morphological pool at the same $k$ and $K$. A second ablation isolates the constraint: adversarial selection within a statistical pool truncated to the same $K$.
Why not just…?
Why not just use BPE-dropout? It is the most serious threat, because it is nearly free, and Tokenization Falling Short shows it helps on character-level tasks. But random drops spend most of their perturbations on splits that neither noise nor inflection produces. ADVSR already reports that adversarial selection keeps a wide lead over random subword regularization under character noise from 0.1 to 0.5. The equal-wall-clock comparison will settle whether that lead survives once dropout gets our compute. If it does not, the proposal reduces to "use dropout", and we will report that.
Why not just deploy a morphology-aware tokenizer? The Spanish results show it improves representation quality. It also pushes the segmenter into inference, where the file notes that upstream tools misparse noisy, informal, or domain-shifted text. MTNT Reddit text is exactly that regime. The static hybrid baseline, evaluated on MTNT and on synthetic noise, will show whether the cascading errors cancel its gains.
Why not just canonicalize the input? For attacks like AdvTok that keep the character string fixed and change only token boundaries, decoding to text and re-tokenizing canonically plausibly neutralizes the attack whenever the defender controls the tokenizer. This is why we do not claim a defense against that threat. Canonicalization does nothing when the string itself changes, as with typos or variants. That is the threat we target.
Why not just use an inference-time defense such as Dynamic Attention? Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention avoids training entirely and reports up to 33% improvement over defensive dropout against TextFooler and PWWS. However, it targets word substitutions that hijack attention, and it was evaluated with standard tokenization on English. We will apply it at test time on top of each trained model to check whether it is complementary or redundant for segmentation noise.
Proposed Experiments
For translation, we use IWSLT Czech→English, a morphologically rich source, and IWSLT17 French→English, with synthetic character insertions, deletions, and substitutions at rates 0.1–0.5, plus MTNT English→French for real user noise, all following the ADVSR protocol. For a second language family and task type, we fine-tune Spanish encoders on XNLI Spanish and PAWS-X Spanish, evaluated clean and under the same character-noise sweep, with MorphAGram segmentation reused from the Spanish tokenization study. Metrics are BLEU or accuracy under noise, clean-accuracy change, and forward-pass equivalents and wall-clock time per epoch.
Four outcomes decide the claim. First, adversarial selection within the morphological pool must beat random selection within the same pool. If it does not, adversarial choice adds nothing. Second, at equal $K$, the morphological pool must match or beat the statistical pool on MTNT. If statistical candidates win on real noise, the "realistic noise lives on morpheme boundaries" hypothesis is wrong. Third, the robustness-per-compute curve must dominate BPE-dropout and ADVSR somewhere on the budget axis. Fourth, the gain over random regularization should be larger for Czech than for French. We will quantify this with measured fertility, since the Token Tax literature The Token Tax predicts high-fertility languages have more fragile segmentations to protect. A null on the fourth check would still leave a useful efficiency result. A failure on the first or third would sink the proposal.
References
- Adversarial Subword Regularization for Robust Neural Machine Translation
- Tokenization Falling Short: On Subword Robustness in Large Language Models
- Adversarial Tokenization
- Exploring morphology-aware tokenization: A case study on Spanish language modeling
- MVP-BERT: Multi-Vocab Pre-training for Chinese BERT
- Towards Improving Adversarial Training of NLP Models
- Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
- The Token Tax: Systematic Bias in Multilingual Tokenization