When Shortcuts and Grammar Disagree: Conflict-Stratified Scoring of Minimal-Pair Benchmarks

A standard way to test whether a language model knows grammar is to show it two nearly identical sentences, one grammatical and one not, and check which one it assigns higher probability. These tests are called minimal pairs. Benchmarks such as BLiMP bundle tens of thousands of them into one accuracy number. Many individual pairs, however, can be passed without any grammar: the correct sentence may also contain the more frequent words, the more familiar local word sequence, or the verb that agrees with whichever noun happens to sit next to it. We propose to label every pair by whether cheap, grammar-free shortcuts point toward the right answer or the wrong one. Models would then be scored on the pairs where shortcuts point the wrong way. We would test whether choosing models and training checkpoints by the usual aggregate score quietly favours the shortcut-reliant ones.

The claim, in two sentences: minimal-pair accuracy is itself an average that can hide the one slice that matters, the pairs where surface statistics and grammar conflict. Making that slice visible should change which models and checkpoints we believe have learned syntax. Robustness research has a name for the selection failure: the "validation trap", where picking checkpoints by average score picks the ones that lean hardest on shortcuts. Syntax evaluation has never measured it.

What is already known, and what is not

Evidence that surface heuristics carry much of the apparent competence comes from agreement studies. Are Transformers a Modern Version of ELIZA? sorted French object–participle agreement sentences by how many of four hand-defined surface heuristics (first noun, last noun, last token, majority number) point to the correct form. A 2-layer LSTM scored 96.4% when the heuristics aligned but only 34.9% on "0-heuristic" sentences, where none of them does; a Transformer reached 74.1% on those. Does BERT really agree? showed that BERT's accuracy on object relative clauses falls toward chance on nonce sentences, which keep the grammar but replace content words with random ones of the same category. It also showed that swapping the word just before the verb matters more than swapping the actual subject.

Both studies are the direct precedent for this work, and both are single-phenomenon analyses with hand-built strata. We generalize their stratification to every pair in a broad benchmark using automatic heuristic labels that do not depend on the model being tested. We then ask a question neither study asked: what these strata do to model ranking and checkpoint selection.

Two papers constrain the design. What Can String Probability Tell Us About Grammaticality? proves that raw string probability mixes how plausible a sentence's meaning is with whether it is grammatical. Pooled discrimination is therefore poor (ROC AUC below 0.75) even when paired accuracy is about 80%, and minimal pairs are the remedy. We accept that argument at the level of a single pair. What we dispute is that averaging over pairs is innocent. A pair cancels the meaning prior, but it does not cancel differences in word frequency or local n-gram familiarity between its two members.

Do Syntactic Probes Probe Syntax? found a similar fragility in a different setting. Probes are small classifiers trained to read syntax out of a model's internal states, and theirs lost an average of 15.4 attachment-score points on Jabberwocky sentences, where content words become pseudowords. That result warns us that removing words also changes the tokens the model sees. Relexicalization therefore has to be controlled, not assumed clean.

The aggregation lens and the selection lens come from outside linguistics. Coarse race data conceals disparities in clinical risk score performance found that performance varied more within coarse groups than between them. That is exactly the test we will apply to BLiMP's 67 paradigms, the benchmark's individual hand-built datasets of 1,000 pairs each. Benchmarks Are Not Atomic showed that reweighting a benchmark's subject mix flips pairwise model rankings nearly 30% of the time, but it covered knowledge and skill axes, not linguistic minimal pairs. Efficient Bias Mitigation Without Privileged Information described the validation trap and a "price of unawareness", the cost of selecting models without knowing the hidden groups. Its evidence comes from image benchmarks such as Waterbirds and CelebA. We carry the concept into language-model checkpointing.

Core Claim

For each minimal pair, a set of grammar-free "voters" can decide whether the pair is shortcut-aligned, shortcut-neutral or shortcut-conflicting. We make three predictions about these labels.

First, conflict-slice accuracy is substantially lower than aggregate accuracy within individual paradigms, not only between phenomena. Second, rankings of models and checkpoints by conflict-slice accuracy diverge from rankings by aggregate accuracy. Third, selecting checkpoints by aggregate accuracy costs measurable conflict-slice accuracy during pretraining.

The voters are fitted only to the pretraining corpus, never to the evaluated model, so each label is a property of the pair rather than of the model. There are four:

  • Local n-gram voters: 2-, 3- and 5-gram models trained on the same corpus as the evaluated model.
  • A frequency voter: it compares the summed log unigram frequencies of the two sentences.
  • An order-blind voter: a bag-of-words scorer. It abstains on pairs that differ only in word order, which is informative in itself.
  • A nearest-noun voter: for agreement pairs, it predicts the verb form that matches the linearly closest noun.

A pair is conflicting when every voter that does not abstain prefers the ungrammatical sentence. It is aligned when every such voter prefers the grammatical sentence.

We also report a second, model-dependent quantity called lexical stability. For each pair we build several variants that replace content words with others of the same category, frequency-binned and with the number of subword tokens matched. Lexical stability is the fraction of these variants on which the model keeps its preference. It separates "correct because of these particular words" from "correct for this sentence frame".

Research Program

The first step is to build and validate the labeler. Before trusting automatic voters on English, we would replicate the French study's hand-defined strata. We train the n-gram and frequency voters on a French Wikipedia corpus of the size used there, 100M tokens, and measure how well automatic conflict labels recover their 0-heuristic category. If agreement is poor, the voter set is wrong and must be revised before anything else proceeds.

The second step is to label BLiMP and SyntaxGym. We label all 67,000 BLiMP pairs (BLiMP) and the SyntaxGym suites (SyntaxGym), with voters trained on each evaluated model's pretraining corpus where it is known. For public models, where it is not, we use a large reference corpus and state this as a limitation.

Slicing makes each group smaller and its estimate noisier. We handle this in three ways. We report bootstrap confidence intervals, set a minimum slice size below which a cell is pooled upward, and fit a hierarchical binomial model that shrinks sparse paradigm×stratum cells toward their phenomenon mean.

The third step is to study selection during training. We train small autoregressive models of 5M to 30M parameters on the 10M-token baby-cosmo-fine corpus used in Learning syntax without semantics. Training covers both natural text and that paper's SAMBAL text, which replaces all content words while keeping function words and morphology. We run several seeds per configuration and save dense checkpoints.

At each checkpoint we score aggregate accuracy and conflict-slice accuracy on a held-out validation half of the labeled benchmark, and report test-half results for the chosen checkpoints. We then compute the price of unawareness:

$$\mathrm{PoU} = \frac{A_{\text{conf}}(c^{\ast}_{\text{conf}})}{A_{\text{conf}}(c^{\ast}_{\text{agg}})}$$

Here $A_{\text{conf}}(c)$ is test conflict-slice accuracy at checkpoint $c$. The checkpoint $c^{\ast}_{\text{agg}}$ is the one chosen by aggregate validation accuracy, and $c^{\ast}_{\text{conf}}$ is the one chosen by conflict-slice validation accuracy. A value of 1.0 means ignoring the conflict slice costs nothing, and values above 1.0 measure the damage.

The SAMBAL arm makes a discriminating prediction. Removing content words should shrink the gap on frequency-voter conflicts, but it should not shrink the gap on nearest-noun and local n-gram conflicts, because those shortcuts rest on word order and function words, which SAMBAL preserves. If the SAMBAL models close both gaps equally, our account of what the voters capture is wrong.

The fourth step is to re-rank public models. We score GPT-2 and Llama-3 (the models used by the string-probability paper), plus BERT-style models via pseudo-likelihood, on aggregate, per-paradigm and conflict-slice accuracy. We then measure rank agreement and pairwise flip rates, using the roughly 30% flip rate from BenchHub as an external reference point.

Baselines

The naive reference point is the benchmark's own headline number: aggregate BLiMP accuracy, for which BLiMP reports 83.0% for GPT-2 against 88.6% for humans. Selecting checkpoints by this number is the "do nothing" policy that the price of unawareness is measured against. It needs no tuning and uses the same validation half as every other policy.

The strongest existing evaluation practice is per-paradigm reporting with worst-paradigm scoring. This means reporting all 67 paradigm accuracies and ranking or selecting by the minimum, which is the natural coarse-subgroup analogue of worst-group accuracy. It is the strongest competitor because it already exposes BLiMP's phenomenon-level spread, including GPT-2's above-99% scores on agreement paradigms. We found no reported per-paradigm worst-case number on Lacuna for any model in this file. We will compute it on the identical checkpoints and data. Our method beats it only if conflict slices produce model rankings or checkpoint choices that differ significantly from the worst-paradigm ones, and if within-paradigm conflict-versus-aligned gaps exceed the between-paradigm spread.

The strongest existing stratified method is the hand-heuristic stratification of Are Transformers a Modern Version of ELIZA?, which reports 34.9% (LSTM) and 74.1% (Transformer) on 0-heuristic French sentences. On their data we are not trying to beat those numbers. The goal is to reproduce the same qualitative gap automatically, with conflict-slice LSTM accuracy near or below chance and a clear Transformer advantage. The automatic voters are only credible if they do.

The key ablation removes one voter family at a time and replaces the voter-defined conflict set with a random subset of matched size. If random subsets produce similar rank changes, the effect is just slicing noise rather than shortcut conflict.

Why not just…?

Why not just report the 67 per-paradigm scores? This is the most serious threat, because much of the "hidden failure" in BLiMP is the gap between agreement and islands, and per-paradigm tables already show it. Coarse disaggregation hid within-group disparities in the clinical setting (Coarse race data conceals disparities), but paradigms are not races, and the analogy has to be earned. The hierarchical model settles it directly by estimating the variance of accuracy across conflict strata within paradigms against the variance across paradigms. If the within-paradigm component is small and conflict rankings match worst-paradigm rankings, we will report that per-paradigm reporting suffices, and that is a useful negative result.

Why not just evaluate on a nonce version of BLiMP? Nonce sentences remove meaning, but they keep linear order and function words, and those carry heuristics. The French study found LSTM collapse on heuristic-conflicting cases, and BERT remained sensitive to the adjacent word in nonce frames (Does BERT really agree?). Nonce substitution also changes subword tokenization, which is one plausible reason probes lost 15.4 points on Jabberwocky (Do Syntactic Probes Probe Syntax?). We include nonce aggregate accuracy as a baseline. The SAMBAL prediction above tests directly whether meaning removal alone closes the conflict gap.

Why not just define the hard set as the pairs a weak model gets wrong? This is how robustness methods find hidden groups without labels, and it could plausibly work as well as our voter-defined conflict set. Our 5-gram voter is essentially this idea, and BLiMP reports that the 5-gram model performs poorly across the board. That means a weak model's errors mix pairs whose shortcuts point the wrong way with pairs that are merely hard, so the error set may not isolate shortcut conflict. We will compare the two definitions head to head on price of unawareness and rank divergence. If the weak-model error set matches the full voter set, we will recommend it as the cheaper tool and keep the named voters only as an interpretation layer.

Why not just rely on scale? Overparameterization hurts worst-group accuracy with spurious correlations shows that more capacity can make worst-group accuracy worse, while the French study found the Transformer far better than the LSTM on conflict cases. The public-model re-ranking tests this directly: if the conflict gap shrinks to zero with scale, the result is confined to small models, and we will say so.

Decisive Checks

The project succeeds only if four things hold. First, the automatic labeler recovers the French study's hand-defined strata. Second, within-paradigm conflict gaps exceed between-paradigm spread on BLiMP, with confidence intervals excluding zero. Third, the price of unawareness in the small-model pretraining runs exceeds 1 across seeds. Fourth, conflict-slice rankings of public models differ from both aggregate and worst-paradigm rankings beyond what matched random slices produce.

The project fails cleanly if per-paradigm worst-case scoring or the weak-model error set reproduces every one of these effects. In that case the practical recommendation is simply to use them.