Cheap Invariance Instead of Adversarial Training for Cross-Lingual Prompts

Models trained on English data and then used directly in other languages lose accuracy in two ways. Some of the loss comes from overfitting: the model latches onto English-specific patterns, including the artifacts of any synthetic training data. Some comes from shortcuts: it leans on names and subwords that the two languages happen to share. The strongest existing recipe for the first problem adds generated training examples and then runs adversarial training, a min-max procedure that repeatedly searches for the worst small perturbation of each input and trains against it. That search multiplies the cost of every update. The recipe also needs an external teacher model to catch generated examples whose label has silently changed, and a parallel corpus to reorder English into target-language word order.

We propose to replace all three components with something much cheaper. The multilingual backbone stays frozen. We train only about ten prompt vectors prepended to the input, using an ordinary task loss plus one consistency term that ties each example's representation to that of a perturbed copy. The perturbed copies are built so their label cannot change. The same named entity is swapped, consistently across premise and hypothesis, for an entity taken from target-language monolingual text. Individual tokens are softened toward the backbone's own masked-language-model predictions. Each consistency step costs one extra forward pass instead of a multi-step inner loop. The entity swap aims directly at the shared-vocabulary shortcut. The central question is whether freezing the backbone makes the expensive adversarial step redundant, and if it does not, which part of the benefit is lost.

Core Claim

Data Augmentation with Adversarial Training for Cross-Lingual NLI is the closest prior work. It fully fine-tunes XLM-RoBERTa-base on augmented data. The augmentations are syntactic reordering trained on the JW300 parallel corpus and hypotheses generated by T5, which are then re-labelled by a RoBERTa-Large teacher. Training uses projected gradient descent (PGD), meaning iterated gradient-ascent perturbation of word embeddings. With this recipe it reaches 73.0% average XNLI accuracy, against 69.1% for vanilla XLM-R. Its own ablations say augmentation without PGD overfits to synthetic artifacts. Our claim is that this conclusion depends on the full fine-tuning regime.

Two findings support that reading. Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More tunes only soft prompts on frozen BLOOM and XGLM. It finds that freezing improves transfer to distant languages and weakens the dependence of accuracy on linguistic distance. In other words, the frozen geometry already resists distortion. Robustness Tokens trains ten prefix tokens on frozen vision transformers with a dual clean-invariance and adversarial-correction loss. It shows that a tiny prefix can absorb a robustness objective within 200 steps.

We combine these into one hypothesis. On a frozen multilingual backbone, a consistency loss over label-preserving perturbations recovers most of PGD's transfer gain at a fraction of its compute. It also does something PGD does not: it reduces the collapse that Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets observed when shared entities are replaced by target-language-only ones. We expect worst-case embedding noise to remain the better defense against small contextual substitutions. The proposal is designed to measure that trade-off, not to assume it away.

Research Program

The trainable parameters are $m$ prompt vectors $P$, with $m$ between 10 and 20 following the length range reported for SIB-200. They are prepended to a frozen encoder $f$. For an English training example $x$ with label $y$, we build a perturbed copy $\tilde{x}$ and minimize

$$\mathcal{L}(P) = \ell\big(f(P; x), y\big) + \beta \, \big\| h(P; x) - h(P; \tilde{x}) \big\|^2,$$

where $\ell$ is cross-entropy, $h$ is the pooled final-layer representation, and $\beta$ weights the consistency term. No label is attached to $\tilde{x}$, so a drifted augmentation cannot inject a wrong label. The only way drift can hurt is by forcing invariance where none should exist. We test that failure mode directly; see the decisive checks.

Two perturbation families are used, both chosen to preserve the label by construction.

The first is consistent entity substitution. Every occurrence of a recognized entity in premise and hypothesis is replaced with the same entity of matching type, drawn from target-language monolingual text. Because the replacement is identical on both sides, entailment relations are preserved. No parallel data is needed. The substitution mirrors the P3 vocabulary attack, which removes the shared-entity shortcut.

The second is multilingual text smoothing in the style of Text Smoothing. Selected tokens are interpolated between their one-hot vector and the masked-LM distribution: $\tilde{t}_i = \lambda t_i + (1-\lambda)\,\mathrm{MLM}(t_i)$. Here the masked-LM distribution comes from the frozen backbone itself, so no external model is needed. Text Smoothing was only tested on English, so a secondary question is whether the multilingual masked-LM distribution spreads mass across languages in a way that helps transfer.

We then build a cost-matched ladder of variants on the same frozen backbone and prompt. It runs from plain prompt tuning, to prompts with one-step embedding perturbation, to prompts with $K$-step PGD as in the closest work, to our consistency prompts, to consistency combined with one-step PGD. Every run logs forward-backward passes per update and GPU-hours. Because the backbone is frozen, PGD still has to backpropagate through it. It therefore keeps most of its cost even in the prompt setting, which is why replacing it matters.

One honest risk is that soft prompts are weaker on base-size encoders than on large decoders. We therefore run XLM-R base, to compare against the 73.0 figure, alongside XGLM and BLOOM, to match the SIB-200 study. If prompts on XLM-R base fall well short of full fine-tuning, the comparison will be reported per backbone and not averaged away.

Baselines

The naive reference point is vanilla zero-shot transfer: XLM-R base fully fine-tuned on English MNLI and evaluated on the other XNLI languages. Data Augmentation with Adversarial Training for Cross-Lingual NLI reports 69.1% average accuracy across 15 languages for it. We rerun it under our data pipeline so that differences are attributable to method.

The strongest existing method is the full recipe from that same paper: reorder plus semantic augmentation, teacher rectification, and PGD, reported at 73.0% average XNLI accuracy with XLM-R base. It is the strongest competitor because it is the only cited method that attacks synthetic-data drift and cross-lingual transfer together on this exact benchmark and backbone. We reproduce it with the paper's stated configuration and report our reproduced number next to the published one. We give it the same English training data and the same hyperparameter-search budget, counted in trials, as our method. Its teacher-inference and PGD compute are charged in the cost ledger. Beating it means matching its XNLI average within 0.5 points while using at most one third of its training GPU-hours, with no parallel corpus and no teacher. Any accuracy win at equal cost is a bonus.

For reference, XTREME reports 79.2% XNLI accuracy for XLM-R. The file does not state the model size behind that number, so we treat it only as an upper reference and not as a like-for-like target.

The second strong baseline is frozen-backbone soft prompt tuning as in Soft Prompt Tuning for Cross-Lingual Transfer. We train it on the same augmented data with no consistency term. On SIB-200 there is no reported overall accuracy on Lacuna for this method, so we rerun it with matched prompt length and steps.

The key ablations isolate the consistency ingredient. The first trains on the perturbed copies as ordinary labelled examples, which tests whether invariance matters beyond data volume. The second uses each perturbation family alone. The third replaces consistent entity swaps with independent per-side swaps that can break entailment, which tests whether label preservation by construction matters.

Why not just…?

Why not just prompt-tune on the augmented data and skip any robustness term? This is the most serious threat. Freezing alone may already prevent the artifact overfitting that motivated PGD. If plain prompts on augmented data match consistency prompts on XNLI, our transfer claim reduces to "freezing suffices," and we would report it that way. What would still separate the two is the WikiANN P3 test. Data augmentation adds examples but does not explicitly penalize reliance on shared entities. The ablation that trains on perturbed copies as labelled data settles this directly.

Why not just use single-step adversarial perturbation with full fine-tuning? One-step perturbation roughly halves PGD's overhead and keeps the full-model capacity. The evidence against it as a complete answer is indirect. Full fine-tuning distorts the pretrained cross-lingual geometry, according to the soft-prompt study. Embedding noise also does not target vocabulary shortcuts, according to the robustness-paradox study. Both claims are tested here because one-step full fine-tuning is a rung on our cost ladder. If it wins on accuracy at similar cost, the frozen-prompt argument fails.

Why not just translate the training data (translate-train)? In the closest work, the augmented zero-shot model already beat translate-train in several languages, including Chinese and French. UniPrompt reports translation-based prompts at 34.83% on MARC at 32 shots, against 46.61% for UniPrompt itself, in Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt. Translate-train also needs a translation system, which is exactly what is missing for the low-resource languages where the performance gap is largest. We include it on the high-resource XNLI languages only, as a ceiling-style comparison.

Why not just keep the teacher filter and drop PGD? The filter costs a large-model inference over every synthetic example and carries the teacher's own biases. Our objective never assigns labels to perturbed copies, so it has nothing for a teacher to rectify. The controlled-drift check below shows whether that is sufficient.

Proposed Experiments

Training uses English MNLI, and evaluation is zero-shot on the 15 XNLI languages. We report average accuracy, the gap between English and the target-language average, and GPU-hours per run.

Robustness is tested on WikiANN for the 21 low-resource languages and 13 language pairs of Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets, under its P1–P5 perturbations. The primary metric is the F1 drop under P3. Substitution entities are drawn only from training-split annotations, so test entities never leak into training.

Breadth is tested on SIB-200 with XGLM and BLOOM. There we measure whether the correlation between accuracy and linguistic distance, which the soft-prompt study found that freezing reduces, shrinks further.

The decisive checks are four. First, consistency prompts must match the reproduced full recipe within 0.5 XNLI points at no more than one third of its compute. Second, they must cut the P3 F1 drop relative to plain prompts. A cut in the P4 drop is not required; if PGD stays better on P4, that confirms the expected trade-off.

Third, a controlled-drift experiment. We deliberately corrupt 0%, 10%, and 30% of perturbed copies with entailment-breaking edits and measure how accuracy degrades. This determines whether label-free invariance tolerates drift better than training on drifted copies as labelled data.

Fourth, we hand-audit 200 entity-swapped XNLI pairs to measure the actual label-flip rate of our "label-preserving" construction.

Interpretation of the outcomes is fixed in advance. If plain augmented prompts tie on every check, we conclude that freezing, not invariance, does the work. If PGD prompts win on accuracy at equal cost, the cheap substitute fails and we say so.