Is the Low-Resource Gap Partly in the Answer Key? Native-Speaker Disagreement on Multilingual Minimal Pairs

Multilingual benchmarks report that language models do worse on low-resource languages. The usual reading is that the models know these languages less well. A second explanation has never been tested: the answer keys may also be worse, or more ambiguous, in exactly those languages. This proposal tests it on minimal pairs, which are two sentences that differ in one word, where the benchmark says one is grammatical and the other is not. We will collect graded judgments from several native speakers per item across languages of different resource levels and scripts. Then we ask how much of the reported gap survives once each item is scored against what native speakers actually accept, rather than against a single automatically derived label.

The question matters because the strongest evidence for the gap comes from benchmarks whose labels are built automatically from resources whose quality itself tracks language resource level. If that confound is real, current leaderboards overstate how badly models fail in low-resource languages, and in a way that also misleads model comparisons. If it is absent, we will have shown for the first time that the gap is a property of the models and not of the measuring stick. Either outcome changes how the field reads massively multilingual results.

Core Claim

MultiBLiMP 1.0 extracts over 128,000 agreement minimal pairs in 101 languages from Universal Dependencies treebanks (hand-parsed sentence collections) and UniMorph lexicons (tables of word forms). It reports that accuracy correlates strongly with how much pretraining data a language has. That paper itself names treebank and lexicon fidelity as the bottleneck.

We hypothesize the following. Part of the frequency–accuracy correlation comes from items that native speakers do not reliably endorse. Such items arise from parsing errors, from agreement that is genuinely optional in the language, or from dialect variation. These items should be more common in low-resource languages. The claim is falsifiable in a direct way. The slope of accuracy against pretraining frequency should flatten when accuracy is weighted by native-speaker endorsement, and the flattening should exceed what simpler proxies for item quality produce.

Where This Sits

The motivating evidence comes from English error detection and correction. That literature has shown repeatedly that a single gold label hides real ambiguity:

We borrow the scoring idea from They Can Help. That work weights each test item by crowd consensus, and doing so cut a language model's precision from 0.957 to 0.900. All of this work is English-only. None of it asks whether ambiguity varies across languages in a way that biases cross-lingual comparison.

On the multilingual side, the closest work measures disagreement but stops short of using it to correct model evaluation:

  • Holistic IAA finds within-language annotator agreement around 0.64 versus around 0.49 across languages. It concerns persuasion labels and never re-scores models.
  • How Reliable is Multilingual LLM-as-a-Judge? measures agreement among model judges (average Fleiss' κ ≈ 0.3, a measure of agreement among several raters). It does not compare those judges to human disagreement distributions.
  • MENLO aggregates human preferences at α = 0.84 but does not model variance across native speakers.

One paper constrains our design. MELA treats acceptability as an expert-curated binary property and still finds meaningful cross-lingual signal (GPT-4o at 51.14 MCC). We take this to mean that clean, literature-derived items are not hopelessly ambiguous. We therefore use MELA items as a low-ambiguity control, and we do not assume ambiguity is everywhere.

Our study differs from all of the above in one respect. It is the first to place multiple native-speaker judgments directly onto items of an existing massively multilingual benchmark, and then re-run that benchmark's central cross-language analysis.

Research Program

Language and item selection. We will select about fifteen MultiBLiMP languages. They will span high, mid and low pretraining frequency, Latin and non-Latin scripts, and treebanks of varying size. Several will overlap with MELA, so that curated and automatic items can be compared within one language.

Annotation. Per language we sample about 300 pairs, stratified by phenomenon. For each pair, five native speakers rate both sentences on a graded acceptability scale and indicate which sentence they prefer. We also collect light metadata on region and dialect. This metadata matters because the aggregation model NUTMEG requires it. NUTMEG is a Bayesian model that separates group-specific truths from careless annotators, and its main stated limitation is that corpora lack such metadata. Our data supplies it.

Item scores. Each item gets an endorsement score: the estimated probability that a competent speaker of the relevant variety prefers the benchmark's "grammatical" sentence. Items are also tagged as clear, gradient, dialect-split or mislabeled.

Model evaluation. We score the same open model families MultiBLiMP evaluates, including base and chat variants and the small monolingual Goldfish models. Each is scored three ways: raw accuracy, endorsement-weighted accuracy, and the correlation between the model's log-probability margin and the human preference rate.

Decomposition. The central analysis splits the accuracy gap between high- and low-frequency languages into two parts. One part comes from low-endorsement items. The other comes from items humans agree on. We then re-test MultiBLiMP's two headline secondary findings on the clean items only: that chat models trail base models, and that Goldfish models beat large multilingual models in low-resource languages.

Baselines

Naive reference point. The naive reference is the official MultiBLiMP protocol: raw accuracy against the automatic label, on the same sampled items and the same models. MultiBLiMP reports that the best models average about 90% accuracy. Every reweighted number is compared against raw accuracy on identical items, so any change can only come from the weighting.

Strongest cheap competitor: a metadata proxy. This baseline regresses per-language accuracy on treebank size and annotation source alongside pretraining frequency. It costs nothing and directly targets the confound we claim, which makes it the most dangerous competitor. It uses the same languages and the same model scores. We beat it only if adding endorsement explains a significant amount of extra cross-language variance in accuracy beyond the metadata proxy.

Second competitor: consensus failure across models. This baseline fits an item-response model over the models' own answers and flags items that strong models consistently get wrong as suspect. It is the standard way to find bad benchmark items without humans. We run it on the full model pool with no human data. It counts as beaten if its flags predict human low-endorsement poorly (we measure area under the curve). Poor prediction would mean it confuses hard items with bad ones.

Third competitor: an LLM judge. GPT-4o rates each pair under the same instructions given to the humans. On clean, expert-curated acceptability items it reaches 51.14 MCC on MELA (MELA). It is beaten if its agreement with the human endorsement score drops with resource level more steeply than the human panel's internal reliability does. Fu and Liu leads us to expect that drop. No reported result on Lacuna exists for LLM judges on MultiBLiMP items specifically.

Ablations on aggregation and multiplicity. We aggregate the human judgments three ways: majority vote, MACE, and NUTMEG. NUTMEG's reported correlation with true annotator competence is 0.81, against 0.58 for MACE, on synthetic data. A further ablation isolates our key ingredient, which is having several raters rather than one. In it, each item is checked by a single native speaker. If one verifier reproduces the decomposition, multiplicity is unnecessary, and we will say so.

Why not just…?

Why not just control for treebank size? This is the most serious threat. If treebank quality fully explains the confound, a one-line regression replaces our annotation campaign. But size is a coarse proxy. It cannot tell parse errors from genuinely optional agreement or dialect splits, and only those two need different remedies (fix the label versus accept both forms). The metadata baseline above settles the empirical question directly. If it matches endorsement, our contribution shrinks to validating the proxy, which is still useful.

Why not just drop items that all models fail? Consensus failure across models confounds two things we need to separate. An item might be one where every model shares a real weakness, which MultiBLiMP's frequency effect suggests is common. Or it might be mislabeled. Filtering on model failure would mechanically shrink the gap and erase real model weaknesses. Our comparison against human endorsement measures how often that happens.

Why not just ask GPT-4o? Model judges agree poorly across languages and worse in low-resource ones (κ ≈ 0.3 on average per Fu and Liu). They also reward stylistic surface features (MENLO; Human Feedback is not Gold Standard). Their reliability is therefore weakest exactly where the confound would live. We still run the judge. If it tracks native speakers in low-resource languages, the method scales cheaply, and we would recommend that instead.

Why not just use MELA's curated items? MELA covers ten languages, with labels drawn from the linguistics literature. It is clean but does not reach the low-resource languages where the gap is claimed. Its binary labels also cannot represent gradient acceptability. We use it as the low-ambiguity control, not as a substitute.

Proposed Experiments and Decisive Checks

Primary test. We compare the slope of accuracy on log pretraining frequency under raw versus endorsement-weighted scoring, with confidence intervals from a bootstrap over items and annotators. Our hypothesis predicts two things. First, low-endorsement items will be more common in low-frequency languages. Second, the weighted slope will be significantly flatter, beyond what the metadata proxy accounts for.

Null outcome. If endorsement is uniform across languages and the slope does not change, we report that the MultiBLiMP gap is robust to label quality. This settles the question in the other direction.

Calibration test. For gradient items, we check whether the model's log-probability margin correlates with the share of speakers who prefer each sentence. A positive correlation would show that models encode graded acceptability that binary scoring discards.

Control test. Our endorsement scores should be near ceiling on MELA items in the shared languages. If they are not, our annotation protocol, not the benchmark, is the source of disagreement.

Robustness test. We check whether the base-over-chat and Goldfish-over-multilingual findings hold on items with high agreement. This tells us which published conclusions about multilingual models rest on contested labels.

Release. The released resource, items annotated by several native speakers with dialect metadata, gives the field what Gap 2 of the multi-reference literature lacks: a way to represent more than one acceptable answer across varied grammars.

Main risk. Recruiting five native speakers per item in low-resource languages is hard. We will set a minimum of three raters and report reliability per language, so that thin panels are visible rather than hidden.