Data augmentation means training on deliberately altered copies of each example: characters swapped, words replaced, sentences reordered. It only helps within a band. If the changes are too gentle, the model learns nothing new. If they are too strong, the altered copies stop meaning what their labels say. Practitioners find that band by tuning on a labeled development set. Most of the world's languages have none. Pretraining corpora such as MADLAD-400 cover 419 languages, yet that same paper notes high-quality evaluation sets exist for only about 204 of them. In practice, augmentation strength is tuned in English and copied to every other language. That is reasonable only if the right strength does not depend on the language, and nobody has checked whether it does.
This note proposes to check, and to replace the copy-from-English default with a rule that needs only unlabeled target-language text. The idea in two sentences: perturb raw sentences in the target language at increasing strength, and watch at what strength the model can no longer match each scrambled sentence back to its original. That breaking point measures how much distortion the model tolerates in that language, and we hypothesize it predicts the augmentation strength that works best when fine-tuning on that language.
Core Claim
We make two claims, and the second is only worth pursuing if the first holds. The first is that the best augmentation intensity differs across languages by enough to matter. It should be lower in languages where the model is weak, where the script is not Latin, or where the tokenizer splits words into many fragments. If so, the English-tuned default loses accuracy in exactly the languages that cannot detect the loss.
The second claim is that the per-language optimum can be read off a label-free perturbation curve. Let $R_\ell(\rho)$ be the accuracy with which a frozen multilingual encoder retrieves an original sentence from a pool of 1,000 when given its perturbed version, in language $\ell$ at perturbation rate $\rho$. We set the augmentation intensity for that language as
$$\hat\rho_\ell = \max\{\rho : R_\ell(\rho) \ge \tau\},$$
where $\tau$ is one global threshold, fit on a few languages that do have dev sets and then frozen. The reasoning follows the augmentation overlap theory. That theory argues augmentation helps by linking same-class examples, and hurts once altered views of different classes start to overlap. The point where the model can no longer recognize a perturbed sentence as itself is a natural upper bound on useful distortion.
Relation to Prior Work
The measurement instrument comes from Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes. That paper swaps adjacent characters with probability $\rho$ and scores a 1,000-sentence retrieval task. It reports $r = 0.99$ correlation with XNLI performance and a 0.99 sensitivity threshold below which models fall to chance. It uses one scalar score, on frozen models, only to flag languages a model cannot read at all, and it admits low recall for deeper understanding. We use the whole curve rather than one number, and we use it to set a training hyperparameter rather than to grade a model. That narrower purpose sidesteps the recall problem, because we never claim the curve measures reasoning.
The augmentation side supplies the problem framing but not a usable selection method. DivAug-style diversity-under-affinity selection, entropy-based difficulty selection for text, and InfoTS all balance variety against fidelity. InfoTS does so with a meta-learner that weights transformations end to end. All of them assume a labeled signal or a training objective in the domain at hand, and all treat intensity as language-agnostic. AugGPT frames the same tension as faithfulness against compactness, but only for English.
The closest multilingual work is LLM-powered Data Augmentation for Enhanced Cross-lingual Performance. It compares three fixed prompting pipelines on XCOPA, XWinograd and XStoryCloze, and finds the "Tamil Wall": LLM generations in Tamil were badly ungrammatical. That finding motivates our hypothesis, since it suggests generation fidelity collapses unevenly across languages. But the paper never treats intensity as a continuous knob and never asks whether the optimum shifts by language.
Multi Task Learning For Zero Shot Performance Prediction predicts accuracy from typology, corpus size and tokenizer fertility. It also found fertility drives failures on word-level tasks. This work constrains our design, because it gives a cheaper predictor we must beat. Finally, XTREME-R documents the gap between non-Latin and Latin scripts. That gap is the main reason we expect the optimum to move.
Research Program
We will study three augmentation families, each with a single intensity knob. The first is character neighbor flipping at rate $\rho$. It is the same operation the probe uses, so it gives the most direct test. The second replaces subword tokens with random same-script tokens at rate $\rho$. The third shuffles words locally within a window whose size sets the intensity. To compare families on one axis, intensity is also expressed as the drop in character n-gram overlap (chrF) from the original. A fourth, semantic family is MADLAD machine-translation round trips with sampling temperature as the knob. We will run it only on languages where that translation is usable, because its quality is itself confounded with resource level.
For each language and encoder we compute $R_\ell(\rho)$ on unlabeled text, using the same retrieval protocol as the local-structure paper. We then fine-tune on labeled target-language data across a grid of intensities to find the oracle optimum $\rho^*_\ell$. Three questions follow. Does $\rho^*_\ell$ vary across languages? Does $\hat\rho_\ell$ track it? Does a curve measured with one family transfer to choosing intensity for another? Character-flip curves predicting the best token-substitution rate would show the curve measures general tolerance rather than familiarity with one specific kind of noise.
Baselines
The naive reference point is no augmentation at all: the same fine-tuning data, steps and seeds. Any intensity rule must beat it, or the question is moot.
The strongest realistic competitor is a single intensity tuned on the English dev set and applied to every language. This is what practitioners actually do, and it would be the right answer if the optimum is language-invariant. It gets the same intensity grid, seeds and training budget as our method. Beating it means lower average regret across held-out languages, where regret is accuracy at the chosen intensity minus accuracy at the oracle intensity. The gains must be concentrated in non-Latin and high-fertility languages, and significant over five seeds. There is no reported result on Lacuna for per-language augmentation-intensity selection, so this comparison has no prior number.
The second competitor is a regressor in the style of the zero-shot performance prediction paper, which predicts $\rho^*_\ell$ from typological distance to English, pretraining corpus size and tokenizer fertility. It is fit leave-one-language-out on the same languages used to calibrate $\tau$. That paper reports roughly a 10% MAE reduction from its multi-task Gaussian process in predicting accuracy. That figure concerns predicting performance, not intensity, so it is context only.
The third competitor is the scalar local-sensitivity score from the local-structure paper, used as the predictor in place of the full curve. This isolates whether the curve's shape adds anything beyond one number.
The upper bound is oracle per-language tuning on the target dev set. For context on absolute levels, XTREME-R reports a 65.3 average for XLM-R Large and 54.1 for mBERT. SERENGETI reports an 82.27 average F1 on AfroNLU. The Gen-X paper reports a 12.8-point average XLM-R gain on XWinograd from GPT-4-style augmentation, which shows how much augmentation can matter in this setting.
Why not just…?
Why not just tune once in English and copy the value? This is the real null hypothesis, and the proposal is built to test it. If English-tuned regret stays within seed noise across XNLI and AfroNLU languages, the conclusion is that intensity is language-invariant. We will report that and stop, and that result is itself useful for low-resource practitioners. The script gap in XTREME-R and the Tamil Wall in the Gen-X paper are reasons to doubt invariance, but neither measured it.
Why not just rescale intensity by tokenizer fertility? This is the most dangerous alternative. A character flip in a word that is already split into many fragments plausibly does more damage, so dividing $\rho$ by fertility might capture most of the effect for free. We include fertility normalization, and the typology regressor that contains fertility, as head-to-head baselines. If either matches the curve-based rule, the honest conclusion is that the probe adds nothing beyond a tokenizer statistic. The curve should win only where fertility and model competence come apart, such as low-resource languages written in Latin script, and we will report that subset separately.
Why not just use translate-train data or a bigger model? Translation changes what the training data contains, not how strongly it should be perturbed. XQA shows translation artifacts shift task difficulty, so translated data plausibly needs its own intensity setting rather than removing the need for one. On scale, XTREME-R shows the 13B mT5-XXL losing to XLM-R on retrieval and XCOPA plateauing near 70% against roughly 97% for humans. We test scale directly by comparing XLM-R base with large: if the regret from English tuning vanishes at the larger size, scale is the answer.
Proposed Experiments
The testbeds are XNLI, which has human dev and test sets in 15 languages, and AfroNLU, which has 20 datasets over 28 African languages and supplies the genuinely low-resource, many-script cases. On XNLI we fine-tune on a fixed-size translated training subset per language. On AfroNLU we use the provided training data. Labels in target-language dev sets are used only to compute oracles and regret, and are hidden from every selection rule. Encoders are mBERT, XLM-R base and large, and SERENGETI for AfroNLU. The threshold $\tau$ and the regressors are fit leave-one-language-out.
The first decisive check is the distribution of English-tuned regret across languages. A median below half an accuracy point, with no relationship to script or fertility, falsifies the first claim. The second check is the Spearman correlation across languages between $\hat\rho_\ell$ and $\rho^*_\ell$, compared against the scalar probe, fertility normalization and the typology regressor. The third is cross-family transfer: curves measured with character flips choosing the best token-substitution and shuffle intensities. Success means the curve rule cuts mean regret relative to English tuning, beats the fertility rule on languages where fertility and competence diverge, and holds up across families.
References
- MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
- An Augmentation Overlap Theory of Contrastive Learning
- Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
- Time Series Contrastive Learning with Information-Aware Augmentations
- AugGPT: Leveraging ChatGPT for Text Data Augmentation
- LLM-powered Data Augmentation for Enhanced Cross-lingual Performance
- Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models
- XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
- XQA: A Cross-lingual Open-domain Question Answering Dataset