Most multilingual models are trained on text from more than 400 languages, but there are trustworthy test sets for only a small fraction of them. The cheapest checks we have for the rest are blind to sentence structure. Overlap metrics such as BLEU, which count shared short word sequences ("n-grams") with a reference translation, can give a scrambled sentence the same score as a fluent one. The same metrics punish a correct sentence that simply uses a different legitimate word order than the single reference. The best reference-free check for low-resource languages swaps only adjacent characters, so it cannot see whether a model knows how words and phrases are ordered in that language.
We propose to build word-order minimal pairs automatically for about 90 languages. A minimal pair is two versions of a sentence that differ in one controlled way. Here, every Universal Dependencies treebank sentence is rearranged by moving whole phrases. Each rearrangement is labeled permissible if the treebank shows the language actually uses that order for that construction, and illicit if the order essentially never occurs. Both versions contain exactly the same words, so the test measures order knowledge alone. One resource then supports two measurements. It is a stress test that shows, language by language, when BLEU, chrF, embedding similarity and the older reordering metrics accept illicit orders or reject permissible ones. It is also a reference-free probe of whether a model has learned a language's grammar beyond its character patterns. We expect this probe to explain differences between languages that existing probes treat as already "understood."
Core Claim
A model's knowledge of a language shows in two behaviors. It should reject phrase orders the language does not allow, and it should not reject orders the language does allow. The second behavior has never been measured across many languages. We claim that the gap between the model's score for permissible and illicit rearrangements is a useful unsupervised diagnostic, provided the gap is measured against the language's own continuous word-order flexibility rather than a coarse label like "SOV." This diagnostic should predict downstream task performance among languages that pass existing local-structure probes. The same pairs should also show that n-gram and embedding metrics fail in opposite directions depending on how free a language's word order is.
What Prior Work Establishes and Leaves Open
The motivation comes from two literatures that rarely meet. On the metric side, Re-evaluation the Role of Bleu in Machine Translation Research showed that many nonsensical permutations receive the same BLEU score as a coherent translation. A Quantitative Analysis of Reordering Phenomena explained why: a long-distance reordering is penalized only at the few joints where moved blocks meet. Work on Turkish and Turkmen (Machine Translation between Turkic Languages), Czech (Tackling Sparse Data Issue in Machine Translation Evaluation) and Farsi (Benchmarking SMT Performance for Farsi Using the TEP++ Corpus) documented the opposite failure: legitimate free-order variants are scored as errors. The fixes, LRscore from Reordering Metrics for MT and the rank-correlation metrics of Automatic Evaluation of Translation Quality for Distant Language Pairs, add a permutation-distance term against a reference. They therefore assume both a gold reference and that the reference's order is the only correct one. Our pairs test that assumption directly. These papers supply the metrics we stress-test; they are not competitors for the probe.
On the multilingual side, Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes is the closest and most important prior work. It scrambles adjacent characters in raw text and checks whether a model's sentence representations can still match each original to its scrambled version. This score, Monolingual Local Sensitivity (MLS), correlates with task performance across 44 languages, and languages below 0.99 perform at chance. The authors also note the probe's weakness: a high score does not guarantee understanding. They leave word- and phrase-level order unexamined. Demystifying Neural Language Models' Insensitivity to Word-Order shows, in English only, that task performance tracks local character overlap (rank correlation often above 0.80) far more than how far tokens move (0.10 to 0.40). That suggests character-level probes saturate for languages a model handles reasonably well. In apparent tension, On the Importance of Word Order Information in Cross-lingual Sequence Labeling finds that multilingual models overfit to English word order and that removing position information helps zero-shot transfer. Our probe separates these two stories by measuring insensitivity to illicit orders and oversensitivity to permissible ones as separate quantities, per language.
Two papers supply the method. Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering reorders whole dependency subtrees (a word plus everything that depends on it, so a phrase moves as a unit) using estimated pairwise order distributions. We reuse subtree movement but run it inside a single language and for evaluation, not augmentation. Multilingual Gradient Word-Order Typology from Universal Dependencies extracts continuous order proportions from 132 treebanks covering 91 languages. We use those proportions to label rearrangements and to place each language on a flexibility axis.
Two benchmarks define what is not yet covered. PAWS-X tests meaning-changing word swaps but covers seven languages, relies on machine-translated training data, and is framed as supervised classification. MultiBLiMP 1.0 shows that minimal pairs built automatically from treebanks scale to 101 languages, but it tests only agreement, never order. Our resource is the order-focused counterpart to MultiBLiMP. It differs from PAWS-X in having a permissible class, in being built from native treebank text rather than translation, and in scoring by likelihood rather than training a classifier.
Building the Pairs
For each treebank sentence we consider movable subtrees attached to the main verb and to nominal heads: arguments, adverbials, adjectival and genitive modifiers, and relative clauses. Each candidate move produces a new linear order. For every dependency relation affected by the move, we look up how often the treebank orders that head and dependent in the new direction. A rearrangement is permissible only if every changed head–dependent direction is attested above a frequency threshold. It is illicit only if at least one changed direction falls below a near-zero threshold. Borderline cases are discarded. We sweep the thresholds and report results across the sweep, because frequency is not grammaticality and the labels will be noisy in a predictable way.
Two cheaper families serve as controls built from the same sentences. One is full random word shuffles. The other is the character neighbor flips used by MLS. Because each pair contains the same words, a causal language model can be scored with summed log-probability, and a masked model with pseudo-log-likelihood. No length normalization is needed, and tokenization differences between languages largely cancel within a pair.
Labels are validated by native speakers on a stratified sample in roughly twelve languages. These include the free-order languages studied in the metric literature (Turkish, Czech, Farsi), the distant-order languages where PAWS-X performance dropped (Japanese, Korean, Chinese), rigid-order European controls, and as many lower-resource UD languages as annotator access allows. Speakers judge each variant as acceptable in some context, marginal, or unacceptable. Label precision against these judgments is reported per language, and languages whose precision falls below a preset bar are excluded from headline results.
Research Program
The first study asks what the reordering metrics and their simpler competitors actually do. We treat the original sentence as the reference and both variants as hypotheses, then score them with BLEU, chrF, LRscore, NSRP, and multilingual embedding cosine similarity. We record two rates per language. The false-negative rate is how often a permissible variant scores below an illicit one. The insensitivity rate is how often an illicit variant scores within a small margin of the original. We plot both against each language's continuous flexibility. We expect metrics with explicit order penalties to reduce insensitivity, but at the cost of larger false-negative rates in flexible languages. The rigidity reports for Turkic, Czech and Farsi predict this, but it has never been quantified across dozens of languages at once.
The second study is the probe. For each model and language we compute the order-discrimination score: the fraction of pairs where the model prefers the permissible variant over the illicit one. We also compute two simpler accuracies, original-over-illicit and original-over-permissible. A high original-over-permissible accuracy combined with a low permissible-over-illicit accuracy signals a model that only memorized the most common surface order. That is exactly the overfitting described for cross-lingual sequence labeling. We also compare base models with their instruction-tuned versions, since MultiBLiMP found that instruction tuning hurts agreement competence. Whether instruction tuning hurts order competence too is an open question our data can answer directly.
The third study connects the probe to real competence. Among the languages where existing benchmarks exist, we ask whether order discrimination predicts downstream performance beyond what MLS, agreement accuracy and pretraining data size already predict.
Baselines
The naive reference point is raw per-language log-likelihood (or perplexity) of the unperturbed treebank sentences under each model. It needs no perturbation design at all. It is computed on the same sentences, with the same models, as our probe. Our probe beats it if order discrimination adds significant explained variance in downstream accuracy after perplexity is controlled for, using a regression across languages with model fixed effects.
The strongest existing reference-free probe is MLS from Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes. Its reported result is a Pearson correlation of r = 0.99 between local structure sensitivity and task performance across 44 languages (XNLI and Tatoeba), with 0.99 as the threshold for being above chance. On the full language range this correlation leaves almost nothing to improve. The fair comparison is therefore restricted to languages scoring above the 0.99 threshold, which is exactly where its authors concede low recall. We will rerun MLS with their protocol (character neighbor flipping, 1,000-sentence retrieval) on the same treebank sentences and the same encoders used for our probe, with identical compute. Beating it means significantly higher rank correlation with downstream accuracy within that subset, judged by a bootstrap over languages.
The strongest agreement-based competitor is per-language accuracy on MultiBLiMP 1.0. The file reports best models averaging around 90%, and no per-language figures were found on Lacuna. It is scored the same way as our pairs on overlapping languages and models. Beating it means order discrimination carries information about downstream performance that agreement accuracy does not.
For the metric study, the strongest prior reordering metric is NSRP^{1/4}. Automatic Evaluation of Translation Quality for Distant Language Pairs reports a Spearman correlation of 0.947 with human adequacy on NTCIR-7 Japanese–English, against 0.515 for BLEU. LRscore is the second reordering metric. Reordering Metrics for MT reports Chinese–English systems tuned on it preferred by human judges 52.5% of the time versus 43.9% for BLEU. Here the goal is not to beat these numbers but to measure where these metrics break.
The key ablation replaces our typology-calibrated pairs with random word shuffles, and separately with subtree moves that ignore the permissible/illicit labels. Same sentences, same models, same scoring. If the ablated probes predict downstream performance as well as the full probe, the treebank calibration is unnecessary and the proposal reduces to "extend MLS to words."
Why not just…?
Why not just extend MLS to word-level shuffles? This is the most serious threat, and it is run as the ablation above. Our expectation is that random word shuffles, like character flips, saturate quickly. The English decomposition in Demystifying Neural Language Models' Insensitivity to Word-Order shows models detect damage to local structure easily, and full shuffles destroy local structure. A word-shuffle probe should therefore approach ceiling for most languages that pass MLS. If it does not, and it predicts downstream performance as well as the calibrated pairs, we will report that and drop the calibration claim. The metric-failure analysis would still stand, since it needs the permissible class by construction.
Why not just use chrF or multilingual embedding similarity as the order check? The file already shows why this falls short. chrF keeps most character n-grams intact when phrases are shuffled (Demystifying Neural Language Models' Insensitivity to Word-Order). Averaged embeddings have no order mechanism (WEmbSim). The bag-of-words baseline trailed mBERT by 32% accuracy on PAWS-X. Rather than assume this carries over, our first study measures these metrics' insensitivity rates on every language.
Why not just machine-translate PAWS or the English perturbation suites into more languages? Translation carries English word order and adds artifacts (XQA). PAWS-X itself reports entity noise in its translated data, and MADLAD-400 reports domain skew and translationese for low-resource targets. Translation also cannot produce a language's permissible alternative orders, which are the whole point of the false-negative measurement. Our pairs start from native treebank text instead.
Why not just rely on perplexity? Perplexity is a fair cheap baseline and is included. But it is hard to compare across languages because of tokenization, and a model can assign good likelihood to common local patterns without knowing which phrase orders the grammar allows. The regression described in the baselines will show whether that distinction matters for downstream performance.
Proposed Experiments and Decisive Checks
Pairs are built from the treebanks covered by the gradient typology dataset, about 90 languages. Models are mBERT and XLM-R (the XTREME-R baselines, reported there at 54.1 and 65.3 average) plus open base/instruction-tuned LLM pairs. Downstream targets are XNLI and Tatoeba (as in the MLS study), the XTREME-R tasks, PAWS-X for its seven languages, and MultiBLiMP for agreement. The metric study uses the pairs themselves, plus a check on WMT 2009 human rankings to confirm that our LRscore and NSRP implementations reproduce sensible behavior.
Three outcomes decide the proposal. First, label quality: permissible/illicit labels must reach acceptable precision against native-speaker judgments in most validated languages. Otherwise the resource is not trustworthy, and we would retreat to illicit-only pairs. Second, added information: among languages scoring above 0.99 on MLS, order discrimination must correlate with downstream accuracy significantly better than MLS, and better than the random-shuffle ablation. If it does not, the core claim fails. Third, typology dependence: if the account is right, metric false-negative rates should rise with a language's measured flexibility, and model oversensitivity to permissible orders should be largest in flexible languages that are distant from English. That would settle, at least for the languages tested, whether multilingual models are too insensitive or too sensitive to word order: the answer would depend measurably on the language.
References
- Re-evaluation the Role of Bleu in Machine Translation Research
- A Quantitative Analysis of Reordering Phenomena
- Machine Translation between Turkic Languages
- Tackling Sparse Data Issue in Machine Translation Evaluation
- Benchmarking SMT Performance for Farsi Using the TEP++ Corpus
- Reordering Metrics for MT
- Automatic Evaluation of Translation Quality for Distant Language Pairs
- Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
- Demystifying Neural Language Models' Insensitivity to Word-Order
- On the Importance of Word Order Information in Cross-lingual Sequence Labeling
- Improving Cross-lingual Transfer through Subtree-aware Word Reordering
- Multilingual Gradient Word-Order Typology from Universal Dependencies
- PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
- MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
- WEmbSim: A Simple yet Effective Metric for Image Captioning
- XQA: A Cross-lingual Open-domain Question Answering Dataset
- MADLAD-400: A Multilingual And Document-Level Large Audited Dataset