Multilingual language models are trained on text in hundreds of languages, but we can only check their grammar in about a hundred. For most of the long tail we cannot answer a basic question: does the model actually track grammatical agreement in this language (for example, matching a verb to a plural subject), or does it only produce plausible-looking strings? We propose to answer it using nothing but raw text. Interpretability work has found a single direction inside a model's hidden states that encodes "the subject is plural." Adding that direction, learned from English, makes the model inflect Spanish verbs for plural (On the Similarity of Circuits across Languages). Our idea is to push that English-derived direction into unlabeled text in any language and measure one thing: does the model respond by changing word forms in small, inflection-like ways (era → eran, Swahili m- → wa-), clearly more often than it does when pushed along a random direction of the same size?
If the model has a working agreement mechanism that it shares across languages, the push should cause small, targeted morphological edits. If the language is effectively noise to the model, or the model has no shared mechanism for it, the push should produce either nothing or arbitrary token swaps. The resulting score, which we call minimal-edit steering response, needs no treebank, no morphological lexicon and no annotator. That matters because the existing large-scale grammar benchmark requires both a treebank and a morphological lexicon for every language it covers.
Core Claim
We claim that minimal-edit steering response predicts a model's per-language accuracy on agreement minimal pairs. A minimal pair is two sentences that differ only in one grammatical feature, one grammatical and one not; the model passes if it assigns higher probability to the grammatical one. We further claim that the steering score carries information that existing label-free signals miss. Those signals are character-level perturbation sensitivity, tokenizer fertility (how many pieces the tokenizer splits a word into), pre-training corpus size, and typological distance from English.
The second claim is the point of the proposal. The best existing label-free probe, Monolingual Local Sensitivity from Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes, swaps adjacent characters and measures how much the hidden states shift. Its own authors note that it has high precision for detecting languages the model cannot read at all, but low recall for higher-level competence. It probes only local character order, not hierarchical agreement. Our probe targets exactly the gap they name.
Because the probe only works where agreement machinery is shared across languages, running it across around 100 languages also gives the first large-scale map of where shared agreement mechanisms exist. Until now this has been shown only for English–Spanish in Gemma 2B (On the Similarity of Circuits across Languages) and for five European languages in XGLM (Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models).
Relation to Prior Work
Two lines of work motivate this proposal by documenting the gap. MultiBLiMP 1.0 builds over 128,000 agreement minimal pairs in 101 languages from Universal Dependencies treebanks and UniMorph inflection tables. It finds that performance "varied drastically across languages." Its coverage, however, stops wherever those resources stop. Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models hand-builds test suites for Basque, Hindi and Swahili. It shows that Swahili noun-class agreement sits near chance (average 0.504), while Hindi split ergativity is at 0.873. Neither paper can say anything about languages without hand-built resources. We use both as ground truth and do not replace them.
Three papers supply the method. The Gemma 2B study gives us the procedure for finding a one-dimensional number direction and steering with it at a middle layer. The XGLM/mBERT neuron study gives us causal mediation analysis, which measures how much individual neurons drive the model's agreement preference. Latent Mechanisms of Language Control shows that interventions on hidden features can control output language. We borrow its idea of keeping the language fixed while editing other properties of the output.
The mBERT result is also a constraint, and it could undercut the proposal. In masked models, agreement circuits are disjoint across languages. In such a model, a language could be handled competently by language-specific machinery and still show no steering response. We do not hide this. We treat it as a prediction: the probe should be valid for autoregressive models with shared circuits (Gemma, XGLM) and should break down for mBERT. If it does not break down, that would itself contradict the earlier finding and deserve follow-up.
The performance-prediction line, Multi Task Learning For Zero Shot Performance Prediction, treats models as black boxes. It predicts scores from typological distance, corpus size and fertility, and reports a 10% reduction in mean absolute error, or 20% on tasks with little data. It never looks inside the model. We add an internal, causal feature to that kind of predictor and test whether it earns its place.
Research Program
Extracting the direction. For each model we take English, plus Spanish as a check, subject-number minimal pairs. Following the Gemma procedure, we find the number direction as the difference of mean hidden states between plural-subject and singular-subject contexts at the layer where steering most strongly flips verb predictions. We then confirm causally that adding it flips English and Spanish verbs. The direction is fixed from then on; no target-language data ever touches it.
Scoring a language. For a target language we take 1,000 raw sentences from the same MTData pool used for local sensitivity. At every token position we add the direction, scaled by $\alpha$, to the residual stream, which is the running hidden-state vector the model updates layer by layer. We then record whether the top predicted next token changes. A change counts as "minimal" if the new token shares a stem with the old one, meaning a normalized character edit distance below a threshold, with prefix or suffix edits allowed so that both Bantu prefixing and Romance suffixing count.
The score is
$$S = \Pr[\text{minimal flip} \mid \text{number direction}] - \Pr[\text{minimal flip} \mid \text{random direction}],$$
averaged over 20 random directions with the same norm and over a small grid of $\alpha$. The subtraction removes generic fragility: a model that confuses look-alike tokens whenever its hidden states are pushed will score high on both terms, and the difference cancels it.
A second variant, $S_{\text{neg}}$, steers toward singular and checks that the flips are reversible. A flip that goes one way when pushed toward plural and back when pushed toward singular is evidence of structured inflection rather than arbitrary disruption.
Validating the score. We correlate $S$ with per-language accuracy on MultiBLiMP and on the Basque, Hindi and Swahili suites. We use Gemma-2-2B and XGLM as the main subjects and mBERT as the expected failure case, steering at the masked position for mBERT. As a sanity check, we compare the scores to native-speaker judgments of whether the flipped forms are real inflections in a small sample of languages.
Deployment. We score the roughly 350 MTData languages, including those outside UD. For four or five typologically distant languages with no MultiBLiMP coverage, we commission small native-speaker minimal-pair sets (around 200 pairs each) as true out-of-sample checks.
Baselines
Reference points. The naive reference is pre-training token count alone, together with the predictor that assigns every language the cross-language mean accuracy. The best MultiBLiMP models average about 90% (MultiBLiMP 1.0), so a constant predictor near the per-model mean is the floor to beat.
Monolingual Local Sensitivity. This is the strongest label-free competitor (Detecting Languages Unintelligible to Multilingual Models). It also uses only raw text, and it reports Pearson $r = 0.99$ with XNLI and Tatoeba accuracy. We rerun it on the identical 1,000 sentences per language, with the same models and its published perturbation rate. That $r$ is against semantic and retrieval tasks, not agreement; we found no reported result on Lacuna for local sensitivity versus MultiBLiMP accuracy. Beating it means two things. First, $S$ must correlate with MultiBLiMP accuracy more strongly than local sensitivity does, across languages that both mark as intelligible (score at least 0.99). Second, $S$ must keep a significant partial correlation after controlling for local sensitivity.
Surface-feature performance predictor. We fit a Gaussian-process regressor in the style of Multi Task Learning For Zero Shot Performance Prediction, using URIEL distance, corpus size and fertility. The paper reports a 10% overall MAE reduction over its own baselines; we found no reported result on Lacuna for predicting MultiBLiMP accuracy. We train it under leave-one-family-out cross-validation, with the same splits and the same hyperparameter search budget, with and without $S$ as an added feature. A win is a statistically reliable drop in held-out-family MAE when $S$ is added.
Ablations. Three ablations isolate the key ingredient. The first replaces the causally validated direction with a correlational linear-probe direction for subject number, which tests whether causal validation matters. The second drops the minimal-edit filter and counts any flip, which tests whether the morphological filter matters. The third uses a direction extracted for an unrelated property (sentiment) under the same normalization, which tests whether any meaningful direction would do.
Why not just…?
Why not just build MultiBLiMP-style minimal pairs for more languages? Where treebanks and UniMorph exist, this is strictly better, and we use it as ground truth. The pipeline cannot run where those resources are absent, and that is most of the 350 languages in question. Even the three hand-built suites took curated lexicons and native-speaker validation per language (Controlled Evaluation of Syntactic Knowledge). Our probe is meant only for languages beyond that boundary.
Why not just use local sensitivity? It may be enough, and this is the main threat. If agreement competence simply follows from basic intelligibility, $S$ will add nothing. The partial-correlation test settles this. The case for $S$ rests on the Swahili result: Swahili is presumably intelligible to these models, yet its agreement accuracy is near chance. If $S$ separates intelligible-but-ungrammatical languages like Swahili from intelligible-and-grammatical ones like Hindi, and local sensitivity does not, the probe earns its place. If it fails to, we will report that.
Why not just regress on corpus size, fertility and typology? These features are cheap, and SHAP analysis shows that fertility dominates word-level tasks. The evidence against relying on them alone is indirect but real. Small monolingual Goldfish models beat large multilingual ones on MultiBLiMP despite far less data, and Lao has substantial pre-training presence yet is unintelligible to models. Either result suggests that quantity features miss something. The leave-one-family-out experiment with and without $S$ is the direct test.
Why not just translate English agreement tests? Agreement minimal pairs do not survive translation. The violated feature, its morphological marking and even its presence differ across languages. Machine-translated benchmarks also introduce translation artifacts (XQA), and these would contaminate exactly the forms being tested.
Decisive Checks
The proposal fails if any of the following holds for Gemma-2-2B and XGLM. The first failure condition is that the rank correlation between $S$ and MultiBLiMP accuracy under leave-one-family-out is weak. The second is that the correlation disappears after controlling for local sensitivity and fertility. The third is that the random-direction or sentiment-direction ablations score as well as the number direction. The fourth is that native speakers judge most "minimal flips" not to be real inflections.
We have two specific predictions. On the targeted suites, $S$ should rank Hindi above Basque above Swahili, matching the reported 0.873, 0.741 and 0.504. For mBERT, $S$ should correlate poorly with accuracy, consistent with its disjoint circuits; if it correlates well, the disjoint-circuit account needs revisiting.
Success would yield a label-free agreement screen for long-tail languages. It would also yield a 100-language map of where cross-lingual agreement machinery is shared, testing it well beyond the closely related Indo-European languages studied so far. The four or five commissioned languages provide the final out-of-sample verdict.
References
- On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Task
- Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
- Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models
- MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
- Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models
- Latent Mechanisms of Language Control in Multilingual Language Models
- Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models
- XQA: A Cross-lingual Open-domain Question Answering Dataset