Machine translation systems that score well on clean news text break on the typos, slang and odd punctuation of real users. They also break on deliberately crafted inputs that were optimized against one model and then reused against another, including commercial engines. The usual fix is adversarial training, which means retraining the whole model on worst-case inputs. It is expensive for translation models, and it often lowers quality on clean text. This note proposes a cheaper route. We keep the trained translation model completely frozen. We learn only a handful of extra input vectors, a "prefix", placed before every source sentence. The prefix is trained so that the encoder's internal representation of a noisy or attacked sentence stays close to the frozen model's representation of the clean sentence. The encoder is the half of the model that reads the source.
Two things make this worth testing now. First, the training signal involves only the encoder, and nothing is decoded or backpropagated through the decoder. So the costly inner search for worst-case inputs gets much cheaper than full adversarial training. Second, attacks that transfer between models are thought to work because related models share internal geometry. A defense that reshapes encoder geometry is therefore aimed at the mechanism of transfer, not at one attack. We also test the claim that makes such prefixes attractive, namely that a secret prefix acts like a private key. We test it against the attacker who matters in practice: one who clones the defended system through its API and attacks the clone.
Core Claim
A small learned encoder prefix (about 10 vectors) on a frozen encoder-decoder translation model can match full-model noise training on natural social-media noise. It should keep clean BLEU within noise of the undefended model, and it should cut black-box transfer attack success. All of this should come at a small fraction of the compute of embedding-level or manifold adversarial training.
The method borrows directly from Robustness Tokens. That work trained 10 prepended vectors on a frozen Vision Transformer with two losses: an invariance loss that keeps clean features unchanged, and an adversary-correction loss that maps attacked features back to clean ones. On DINOv2-G it recovered attacked ImageNet accuracy from 0.3% to 63.1%, with clean accuracy at 85.2% against 85.6%. That work never touched discrete text, sequence-to-sequence models, or the transfer threat to its own secrecy assumption. Those three gaps are what this note addresses.
Research Program
Let $E(x; P)$ denote the frozen encoder's output states for source sentence $x$ with prefix $P$, where $P = \varnothing$ means the original model. For the states at the original (non-prefix) positions, we pool subword states within each word. This lets clean and noisy versions of a sentence be compared word by word even when noise changes the subword split. Let $\tilde{x}$ be a noisy version of $x$ and $\delta$ a perturbation of the input embeddings bounded by $\epsilon$. The prefix is trained with
$$\min_P \; \big\|E(x;P) - E(x;\varnothing)\big\|^2 + \lambda \max_{\|\delta\|\le\epsilon} \big\|E(\tilde{x}+\delta;P) - E(x;\varnothing)\big\|^2 ,$$
where $\lambda$ trades robustness against clean fidelity. The first term is the reason clean quality should survive: the decoder keeps seeing nearly the states it was trained on.
We draw the noisy $\tilde{x}$ from a mixture of three sources. The first is natural noise from the Wikipedia-edit and learner corpora used by Synthetic and Natural Noise Both Break Neural Machine Translation. The second is monolingual MTNT Reddit text. The third is word substitutions from a masked language model in the style of Crafting Adversarial Examples for Neural Machine Translation. We use a mixture because that paper showed training on one noise type fails on others.
The inner maximization takes a few gradient steps through the encoder only. The expected per-step cost is therefore a fraction of one full training step. Effective Adversarial Regularization for Neural Machine Translation and Manifold Adversarial Augmentation instead run their perturbations through the full translation loss, and the latter reports 72 hours on 4 V100s for Chinese-English. We measure the actual wall-clock ratio rather than assert it.
The prefix states are withheld from the decoder's cross-attention. The prefix therefore acts only by reshaping how the encoder reads the real tokens. This keeps its effect interpretable and avoids giving the decoder extra memory to hallucinate from.
The second half of the program is the threat analysis that Robustness Tokens leaves open. Publishing Efficient On-device Models Increases Adversarial Vulnerability and Two Sides of the Same Coin show that models sharing a pretrained lineage share vulnerabilities. A frozen public backbone with a private prefix is exactly such a pair. Model Extraction and Adversarial Transferability shows that cheap API queries yield a clone good enough to launch transferable attacks. That result is for classification, where clones reached 90–99% of victim accuracy. We extend it to translation, using sequence-level distillation from the defended system's outputs. We then ask whether attacks built on that clone get through the prefix.
Baselines
The naive reference is the frozen Marian English-French model with no defense. TransFool uses the same family of model and reports white-box attack success of 60–74% and roughly 67% transfer success to Google Translate for En-Fr. This fixes what "undefended" looks like under attack. For natural noise, MTNT reports a standard model dropping from 33.52 BLEU on news to 21.77 on MTNT. Every method below is scored against this frozen model on identical test sets.
The strongest existing method is Iterative Scheduled Data-Switch Training. It alternates noisy and clean training stages tied by a divergence loss. It reports 34.11 clean BLEU on French-English against 31.76 for a vanilla Transformer, with robustness gains on MTNT at 1%, 5% and 10% noise. It is the strongest competitor because it is the only method in the file that claims both clean gains and noise robustness in translation. Its cost is full-model training with roughly doubled iterations. We reimplement it starting from the same pretrained checkpoint and give it the same noise mixture. We also give it the same hyperparameter search budget (a fixed number of configurations) and report its GPU-hours alongside ours.
The second strong competitor is embedding-level virtual adversarial training on both encoder and decoder, from Effective Adversarial Regularization. It reported 29.15 to 30.13 BLEU on IWSLT test2013 EN→DE. We run it with the same data and search budget.
The cheap strong alternatives get matched compute. These are low-rank adapter fine-tuning of the full model on the same noisy mixture, BPE-Dropout, and a spellchecker in front of the model. We describe them in the next section.
Two ablations isolate the key ingredient. The first trains the same prefix with ordinary translation loss on noisy data instead of the encoder-invariance objective. The second drops the clean-fidelity term. If the first ablation matches the full method, the encoder-invariance idea is not doing the work.
Beating the strongest method means three things at the same time. Clean BLEU must be within 0.3 of the frozen model. MTNT BLEU must be at least equal to data-switch training. And the method must use at most one tenth of its training GPU-hours.
Why not just…?
Why not just fine-tune with low-rank adapters on noisy data? This is the real threat. It is also cheap, and data-driven noise works: Improving Robustness of Machine Translation with Synthetic Noise raised MTNT BLEU from 14.42 to 20.49 with tagged back-translation, and fine-tuning on MTNT raised en-fr to 29.73 in MTNT. Adapters change decoder behavior and risk the clean degradation seen with indiscriminate mixing in Data-Switch Training. They also give no specific reason to resist transfer attacks. That is only an argument, so we run adapters at equal parameter count and equal GPU-hours on every metric. If adapters match the prefix on noise, clean quality and transfer, the method claim fails. What remains is the extraction-attack study, and we would report it as such.
Why not just use BPE-Dropout? It is nearly free and helps with misspellings. Evaluating Robustness to Input Perturbations reports 83.55% against 74.01% robustness for standard BPE on EN-DE. However, it acts only inside word boundaries. It does nothing about valid-word substitutions of the kind WSLS uses to cause over 90% mean percentage decrease. It also requires retraining the model, which the frozen setting forbids. We include it on typo noise, where we expect it to be competitive, and on substitution attacks, where we expect it not to be.
Why not just put a spellchecker in front? Synthetic and Natural Noise found spellcheckers only partly recover Google Translate under natural noise, especially for morphological errors. Word-substitution attacks produce correctly spelled words that pass straight through. We include a spellchecker as a front-end to the frozen model in the evaluation, so its ceiling is measured rather than argued.
Why not just prompt the model or mask attention at inference? Bhagwatkar et al. got large gains from a warning prompt, but only in instruction-tuned vision-language models, and the authors flag it as breakable by adaptive attackers. Plain translation models do not follow instructions. Dynamic Attention needs no training and was tested on translation. We run it as a zero-cost baseline, because it could win on cost if its tuned masking does not drop key words.
Proposed Experiments
The backbone is a pretrained Marian En-Fr and Fr-En model. We train on WMT14 En-Fr source text plus the noise mixture. Evaluation uses four sets: clean WMT14 and IWSLT 2016 test sets, MTNT en-fr and fr-en test sets, the natural-noise IWSLT conditions from Synthetic and Natural Noise (whose charCNN ensemble reached 35.70 noisy and 36.97 clean BLEU in French), and TransFool and WSLS attack sets. We report BLEU, together with the ROBUST and CONSIS scores of Niu et al., which correlate at 0.91–0.98.
For meaning-changing substitutions, consistency rewards insensitivity, which is misleading. There we score round-trip reconstruction, as in WSLS, and measure semantic similarity of the source before and after perturbation. We also count an attack as a success only when the similarity exceeds 0.84, the level TransFool reports.
We run three transfer experiments of increasing strength. The first uses attacks crafted on the public undefended backbone. The second uses attacks crafted on a clone distilled from the defended system's API outputs, at query budgets of 10k, 100k and 1M sentences. The third is a fully adaptive white-box attack with the prefix known, which serves as a lower bound. Cost is reported as training GPU-hours and throughput overhead at inference.
The decisive check is the second transfer experiment. If attacks built on a clone of the defended system recover the undefended transfer success rate of roughly 67%, the prefix's apparent robustness was secrecy, not geometry, and the defense is not deployable. If the clone attacks stay well below that rate, the adaptive white-box attack gets far less than the undefended 60–74%, and clean BLEU holds, the proposal's central claim stands. A second check is the translation-loss prefix ablation. It tells us whether encoder-invariance training, rather than extra parameters, produces the effect.
References
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Synthetic and Natural Noise Both Break Neural Machine Translation
- Crafting Adversarial Examples for Neural Machine Translation
- Effective Adversarial Regularization for Neural Machine Translation
- Manifold Adversarial Augmentation for Neural Machine Translation
- Publishing Efficient On-device Models Increases Adversarial Vulnerability
- Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning
- Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!
- TransFool: An Adversarial Attack against Neural Machine Translation Models
- MTNT: A Testbed for Machine Translation of Noisy Text
- Towards Robust Neural Machine Translation with Iterative Scheduled Data-Switch Training
- Improving Robustness of Machine Translation with Synthetic Noise
- Evaluating Robustness to Input Perturbations for Neural Machine Translation
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design
- Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention