Minimal Pair Evaluation of Language Models

264 concepts191 papers1 proposal

When evaluating how well a neural language model understands human language, standard benchmarks often rely on aggregate metrics like perplexity or downstream task accuracy. However, as noted by James R. Glass and Yonatan Belinkov in Analysis Methods in Neural Language Processing: A Survey, these macroscopic scores fail to provide a granular assessment of a model's true linguistic sensitivity. A model can achieve high accuracy on a benchmark by exploiting shallow statistical shortcuts, local word frequencies, or semantic associations, all without internalizing the underlying structural rules of language.

To look inside the black box of a model's probability distributions, researchers use minimal pair evaluation (also known as targeted syntactic or contrastive evaluation). This paradigm isolates specific linguistic capabilities by forcing models to choose between two nearly identical inputs that differ by only a single feature.

The Confounding Nature of String Probability

To understand why minimal pairs are necessary, we must look at what a language model actually computes. A language model assigns a probability to a sequence of words. In What Can String Probability Tell Us About Grammaticality?, Jennifer Hu et al. formalize a critical challenge: a string's probability is a mixture of its message (the inherent semantic plausibility of the scenario) and its grammaticality (its structural validity).

Consider these two sentences:

  1. The dog chased the cat. (High probability: common message, grammatical structure)
  2. The cat chased the dog. (High probability: common message, grammatical structure)
  3. The house chased the cat. (Low probability: bizarre message, grammatical structure)

If we want to know whether a model understands English grammar, simply checking if it assigns a high probability to a sentence is a noisy strategy. A model might assign a low probability to a perfectly grammatical sentence simply because the message is rare or unexpected. Conversely, it might assign a high probability to an ungrammatical sentence because it contains a cluster of highly frequent words.

The Intuition: Subtracting the Message

Minimal pair evaluation solves this confounding problem by holding the semantic message constant. First popularized at scale by Rebecca Marvin and Tal Linzen in Targeted Syntactic Evaluation of Language Models, the method presents a model with two sentences that are identical in vocabulary and meaning, except for one crucial grammatical modification:

  • Grammatical: The chef that the cooks admire cooks a great meal.
  • Ungrammatical: The chef that the cooks admire cook a great meal.

Because both sentences use the exact same words and convey the same underlying scenario, the semantic "message" component of the probability is identical. When we compare the model's internal preferences for these two strings, the influence of the message is effectively subtracted out. What remains is a pure, isolated signal of the model's sensitivity to a specific structural rule—in this case, long-distance subject-verb agreement.

Quantifying Linguistic Knowledge

Researchers use two primary methods to extract this linguistic signal from a model: forced-choice probability and surprisal analysis.

Forced-Choice Probability

In a forced-choice setup, the model is fed both strings independently. If the model assigns a higher cumulative probability to the grammatical string than the ungrammatical string ($P(\text{grammatical}) > P(\text{ungrammatical})$), it scores a point. This allows researchers to determine if a model possesses a true structural bias or merely relies on linear surface patterns, a question investigated by Tal Linzen in How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech.

Surprisal Analysis

Instead of looking at the whole sentence, surprisal analysis measures the negative log-probability of a specific token given its history. As shown by Jennifer Hu et al. in A Systematic Assessment of Syntactic Generalization in Neural Language Models, a model demonstrates grammatical mastery if its surprisal spikes significantly when it encounters an ungrammatical continuation compared to a grammatical one.

For highly complex, long-distance dependencies, researchers can expand this into a $2 \times 2$ factorial design. For example, Roger Levy in What do RNN Language Models Learn about Filler-Gap Dependencies? crossed the presence of a syntactic licensor (a "filler" word like what) with the presence of its dependent element (a "gap" or missing object). By measuring the interaction of surprisal across all four conditions, the analysis filters out any residual local lexical biases, isolating the model's awareness of abstract structural constraints.

                  [No Gap]                      [Gap]
         +-----------------------------+-----------------------------+
[Filler] | "I know what the lion saw"  | "I know what the lion devoured"
         | (Grammatical)               | (Grammatical)               |
         +-----------------------------+-----------------------------+
[No      | "I know that the lion saw"  | "I know that the lion devoured"
 Filler] | (Grammatical)               | (Ungrammatical)             |
         +-----------------------------+-----------------------------+

Beyond Syntax: Generalizing the Paradigm

The core power of the minimal pair framework is its flexibility. Because it isolates latent variables by holding context constant, the same contrastive mechanism can be applied to evaluate properties far beyond basic grammar.

Social Bias Detection

In CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models, Nikita Nangia et al. applied minimal pairs to measure stereotyping. By presenting pairs where only the token identifying a social group changes (e.g., swapping a historically advantaged group with a disadvantaged group within a stereotypical context), the authors isolated the model's internal demographic biases from the rest of the sentence's structure.

Pragmatics and Discourse Logic

Minimal pairs can also probe whether models understand conversational flow. Allyson Ettinger et al. in Pragmatic competence of pre-trained language models through the lens of discourse connectives constructed pairs where a subtle contextual shift changes which discourse connective (such as but vs. therefore) is logically valid. This isolates deep pragmatic reasoning from simple word-matching heuristics.

Multimodal Alignment

The paradigm even extends to vision-language models. Fei Wang 0032 et al. in Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining? paired a single image with two captions that differed by only one linguistic element (such as word order or negation). If a model cannot reliably match the image to the correct caption over its minimal distractor, it reveals that the multimodal system is operating as a simple "bag of words" rather than processing true compositional semantics.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 214 papers
  2. Version 1 · 2026-10-04 · 460 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_minimal_pair_evaluation_targeted_syntact_df51c481,
  title = {minimal pair evaluation (targeted syntactic evaluation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimal-pair-evaluation-targeted-syntactic-evaluation/rdr_df51c481f5814b8481adb3516e71c4bf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_minimal_pair_evaluation_targeted_syntact_df51c481_v2,
  title = {minimal pair evaluation (targeted syntactic evaluation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimal-pair-evaluation-targeted-syntactic-evaluation/rdr_df51c481f5814b8481adb3516e71c4bf/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake