Nonce MultiBLiMP: Separating Grammar from Vocabulary Across a Hundred Languages

When a multilingual language model passes a grammar test in Swahili or Basque, we usually cannot tell what it knows. It may have learned the agreement rule, for example that a plural subject takes a plural verb ending. Or it may simply have seen these exact words together many times. The largest grammar benchmark covering many languages builds its test pairs from real treebank sentences. It finds that accuracy tracks how much of each language the model saw in training. A smaller study replaced real words with invented ones. It found that training volume did not predict success at all, and that the irregularity of a language's inflection did. These two findings disagree, and the disagreement matters. It decides whether the answer for under-served languages is "add more data" or something else.

We propose to resolve it with a technique borrowed from psycholinguistics, the study of how people process language. The technique is a graded ladder of test stimuli. Each grammatical test sentence from the massively multilingual benchmark is rebuilt at several rungs. The first keeps the real words. The second swaps in other real words of the same type. The third uses invented words that follow the language's sound and spelling patterns. The fourth uses invented strings that break those patterns. The grammatical ending that carries agreement is identical on every rung. How accuracy falls as we climb the ladder gives each language a profile. That profile says how much of the model's apparent grammar is vocabulary memory, and whether the model has learned the language's sound patterns at all. The whole construction runs from raw text and existing treebanks, so it scales to the languages where labeled evaluation is missing.

Core Claim

We claim that a model's score on real-sentence grammar tests contains a hidden term: how familiar it is with the words in the sentence. For low-resource languages that term can be most of the score. We also claim it can be measured separately.

Let $A_k(\ell)$ be minimal-pair accuracy in language $\ell$ at rung $k$ of the ladder. A minimal pair is two sentences that differ only in one grammatical feature; the model is scored correct when it assigns higher probability to the grammatical member. The rungs are $k=0$ for real words, $k=1$ for real-word swaps, $k=2$ for pattern-legal invented words, and $k=3$ for pattern-illegal strings. We define two quantities.

$$\text{lexical reliance } L(\ell) = A_0(\ell) - A_2(\ell), \qquad \text{sound-pattern sensitivity } P(\ell) = A_2(\ell) - A_3(\ell)$$

Lexical reliance is how much the score depends on knowing the words. Sound-pattern sensitivity is how much it matters whether the invented stem looks like a possible word of the language.

Our central prediction concerns how these quantities relate to training data. We expect $L(\ell)$ to grow with the amount of language-$\ell$ text in pretraining. We expect $A_2(\ell)$ to track inflectional irregularity much more than data volume. If both hold, the MultiBLiMP correlation with training frequency and the Wug-test null result are both correct. They would simply be measuring different things.

The fourth rung also gives a diagnostic that needs no labels. Suppose $A_2 \approx A_3$ in some language. Then the model does not distinguish possible from impossible words there, even if a character-scrambling probe says the language is "intelligible."

Relation to Prior Work

Two papers motivate the proposal by exposing the contradiction.

MultiBLiMP 1.0 builds more than 128,000 agreement pairs in 101 languages from Universal Dependencies (UD, a multilingual collection of syntactically annotated sentences) and UniMorph (a database of inflected word forms). It reports that accuracy strongly tracks training frequency and that base models often beat their chat versions. Every item uses real, possibly memorized sentences.

Morphology Matters runs an expert-built Wug test in six languages. A Wug test gives a speaker an invented word, such as "wug," and asks for an inflected form, such as its plural. The paper finds that irregularity predicts errors (β = −0.67) and training volume does not (β = 0.10 and −0.02 for the two models). It covers only two closed models and six languages, five of them Indo-European, so its null result has little statistical power.

Counting the Bugs in ChatGPT's Wugs shows collapse on Tamil and Turkish. It also documents "real-word bias," where the model turns an invented stem into a nearby real word. That is exactly the effect our pattern-legal rung is built to expose, but we measure it by comparing probabilities rather than by prompting a model to generate forms.

The closest existing method is Multilingual Nonce Dependency Treebanks (SPUD). SPUD swaps real content words for other real words that fit the same syntactic frame, in five languages. It finds syntactic probes lose only 3.1 to 8.3 attachment points. We reuse its swap as our first rung and its constraints on keeping inflection intact. We differ in three ways. We add genuinely invented stems. We add an illegal-string control. We extend coverage from five languages to MultiBLiMP's full range.

The illegal rung comes from psycholinguistic work that contrasts real words, "accidental gaps" and illegal syllables. An accidental gap is a form the language could have but happens not to, like English "lat." Mechanisms of Tone Sandhi Rule Application by Non-Native Speakers uses this contrast to separate rule use from dictionary lookup.

Our stem generator is modeled on two sources. One is Wuggy-style generation that follows the letter-to-letter statistics of real words, as used in Comparing Character-level Neural Language Models Using a Lexical Decision Task. The other is n-gram-matched pseudo-Māori, from More than Just Statistical Recurrence. That paper also constrains us: it shows purely statistical tools miss templatic morphology such as reduplication. We therefore restrict the invented-word rungs to suffixing agreement, where the ending can be cleanly separated from the stem.

Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes supplies the strongest label-free competitor. It swaps adjacent characters in raw text and measures how much the model's representations change. Its authors state that the probe is high-precision but low-recall: a low score reliably flags failure, but a high score does not certify competence. Our fourth rung is a structured replacement for its random scrambling.

Research Program

Building the ladder. For each MultiBLiMP pair we align the target word with its UniMorph lemma, to split the stem from the agreement ending. The target is either the subject noun or the agreeing verb. We keep only items where the ending is a separable suffix, and we report per-language coverage honestly. Rung 1 replaces the stem with a real stem of the same part of speech and inflection class, drawn from the same treebank, following SPUD's constraints.

Rung 2 replaces it with an invented stem. The stem comes from a letter-level n-gram model trained on that language's UD lemmas plus MADLAD-400 text, drawn from MADLAD-400. We filter out any string attested as a word in either source. We match each invented stem to the original in length, in number of subword tokens under each evaluated model's tokenizer, and in the average probability of its letter sequences. Rung 3 applies the Local Structure Probe's neighbor-flipping to the invented stem, keeping only outputs the n-gram model scores as very unlikely.

Validation. Native speakers of about ten typologically spread languages rate the rung-2 and rung-3 stems as possible or impossible words. This follows the kind of human validation SPUD used, where 87% of its outputs were judged grammatical. The target is that rung-2 items are rated possible far more often than rung-3 items.

Models. We evaluate base and instruction-tuned pairs of open multilingual models, including Llama-3.1 as used in XCOMPS. We add mBERT and mGPT from SPUD, and the small monolingual Goldfish models that MultiBLiMP found competitive on low-resource languages. Scoring uses summed log-probability for the causal models and pseudo-log-likelihood for the masked ones. This follows XCOMPS' observation that direct probabilities track internal-representation probes.

Baselines

The naive reference point is the unmodified MultiBLiMP score, rung 0, correlated with each language's share of the training data. This is what practitioners currently report. The best reported figure on Lacuna is that the best models reach about 90% average accuracy, with steep low-resource degradation (MultiBLiMP 1.0). Our proposal beats this reference if two things hold. First, the lexical-reliance term $L(\ell)$ must explain a significant share of the variance that training frequency currently explains. Second, the language ranking by rung-2 accuracy must differ materially from the rung-0 ranking.

The strongest existing label-free method is the Monolingual Local Sensitivity probe of Detecting Languages Unintelligible to Multilingual Models. It reports a Pearson correlation of r = 0.99 with XNLI and Tatoeba performance across 44 languages, and a 0.99 sensitivity threshold below which XNLI accuracy is at chance. We run it on exactly the same models and languages using the same raw-text sources, with no tuning on either side. Our ladder beats it if it separates languages the probe leaves undifferentiated. Concretely, among languages above the 0.99 threshold, rung-2 accuracy should still vary widely and should predict held-out MultiBLiMP-style accuracy better than the probe score does.

The key ablation is SPUD-style real-word swapping, rung 1, run on identical items. If rung 1 and rung 2 give the same per-language accuracies, invented words add nothing, and the cheaper swap should be used instead.

The last comparison is the published Wug-test regression from Morphology Matters. We re-estimate the same regression, accuracy on irregularity and on training volume, using our far larger language sample.

Why not just…?

Why not just regress MultiBLiMP accuracy on training data and stop there? That regression cannot tell memorized word pairings apart from learned rules, because both grow with data. The ladder separates them by holding the ending fixed and removing the words. If $L(\ell)$ turns out flat across languages, meaning memorization is not the hidden term, we will report that. It would mean the frequency correlation reflects genuine rule learning.

Why not just use character scrambling, since it already correlates at r = 0.99? That correlation is with coarse downstream tasks. Its own authors concede that passing the threshold does not certify competence. Scrambling also mixes invalid spelling with every higher-level failure at once. Experiment 3 below tests directly whether the ladder adds predictive information beyond the scrambling score.

Why not just swap in other real words, as SPUD does? This is a real threat. Real-word swaps need no phonotactic generator and no human validation. They remove sentence-level meaning but keep each word's familiarity, which is precisely the variable we suspect drives the low-resource gap. The rung-1 versus rung-2 comparison settles the question empirically. If the two agree in most languages, we will recommend the swap and drop invented stems.

Why not just prompt models with Wug tests in more languages? Prompted generation conflates grammatical knowledge with instruction following. MultiBLiMP shows chat tuning itself hurts grammar scores. Counting the Bugs in ChatGPT's Wugs also shows 0% versus about 43% swings on Tamil depending on prompt format. Probability comparison over fixed pairs avoids both problems.

Proposed Experiments and Decisive Checks

The first experiment tests the reconciliation claim. Across all covered languages, we regress rung-2 accuracy and $L(\ell)$ separately on log training share and on an irregularity measure. For irregularity we use UniMorph-derived I-complexity, the measure of how unpredictable inflected forms are that Morphology Matters used. The claim is confirmed if training share predicts $L$ but not rung-2 accuracy. It is refuted, and the Morphology Matters null attributed to low power, if rung-2 accuracy tracks training share as strongly as rung 0 does.

The second experiment checks the illegal-string control. For each language we test whether $P(\ell)$ is reliably positive. Languages where it is near zero are flagged as having no learned sound-pattern knowledge. We compare those flags against the scrambling probe's verdicts.

The third experiment tests predictive usefulness. Holding out MultiBLiMP's real-sentence scores, we ask whether rung-2 accuracy plus the scrambling score predicts them better than the scrambling score alone. We use leave-one-language-family-out evaluation, so the prediction must generalize to unseen families.

The fourth experiment compares base and chat versions of the same model. The question is whether instruction tuning's damage to grammar concentrates on rung 2, which would point to damaged rule use, or on rung 0, which would point to damaged recall of familiar words. Nobody has yet separated these two explanations for the post-training loss MultiBLiMP reports.

Throughout, the matching of subword token counts is essential. Tokenization artifacts would otherwise produce a lexical-reliance signal on their own. We therefore also report every result restricted to items where the original stem and the invented stem are each a single token.