Probing Language Models with Linguistic Minimal Pairs

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2460 papers510 conceptsfrozen 2026-10-04

Standard evaluation metrics in Natural Language Processing, such as accuracy or F1 scores, often provide a misleading sense of a model's true capabilities. A model might achieve state-of-the-art performance on a benchmark by exploiting shallow statistical shortcuts rather than demonstrating a genuine understanding of linguistic structure. To look "under the hood," researchers have turned to minimal pairs—a method borrowed from linguistics to isolate specific variables and measure a model's internal knowledge with surgical precision.

The Logic of Controlled Comparison

In linguistics, a minimal pair consists of two sentences that differ by only a single element, such as a word or a grammatical marker, which results in a change in meaning or grammaticality. In the context of model evaluation, this is often called contrastive evaluation.

As James R. Glass and Yonatan Belinkov describe in their taxonomy of analysis methods, contrastive evaluation moves beyond aggregate scores. By presenting a model with two nearly identical inputs, researchers can determine if the model is sensitive to a specific linguistic feature. For example, Ethan Wilcox and Roger Levy use minimal pair sentence design to test if a model understands subject-verb agreement across long distances. If the model assigns a higher probability to "The keys to the cabinet are..." than to "The keys to the cabinet is...", it demonstrates a grasp of hierarchical syntax that a simple accuracy score on a translation task might hide.

Measuring Knowledge via Surprisal and Logits

The power of the minimal pair approach lies in how it queries the model. Instead of asking a Large Language Model (LLM) to generate text or answer a question—which introduces noise from the model's decoding strategy—researchers often use forced-choice evaluation.

This involves reading the model's internal "logits" (the raw scores before they are turned into probabilities). Jennifer Hu and Roger Levy argue that prompting is not a substitute for probability measurements. They found that models often "know" the correct grammatical form internally but fail to produce it when prompted. By comparing the surprisal—the degree of "unexpectancy" the model assigns to a token—researchers like Emmanuel Chemla can quantify exactly how much a model is "bothered" by a syntactic violation.

Exposing Heuristics and Biases

Minimal pairs are particularly effective at exposing when a model is "cheating." Allyson Ettinger used psycholinguistic diagnostics to show that while BERT is excellent at word associations, it is often "blind" to negation. In a minimal pair like "A robin is a bird" vs. "A robin is not a bird," a model relying on shallow associations might assign high probability to both because "robin" and "bird" are closely linked, failing to realize that the word "not" fundamentally flips the truth value.

This technique extends to social biases and safety. Rachel Rudinger and colleagues developed Winogender schemas, which use minimal pairs to test if coreference systems rely on occupational stereotypes (e.g., assuming a "surgeon" is male). Similarly, Paul Röttger introduced HateCheck, a suite of functional tests that use contrastive cases to see if hate speech detectors are fooled by simple changes, like adding "not" to a hateful slur or using a slur in a reclaimed, non-hateful context.

Competence versus Performance

A recurring theme in this research is the distinction between a model's competence (what it knows) and its performance (what it does). Linyang He and Hinrich Schütze suggest that LLMs are "form-first" learners, mastering grammatical structures (form) earlier and more robustly than they master conceptual meaning.

By using diagnostic probing on minimal pairs, researchers can subtract the neural activations of one sentence from its pair. This process cancels out the shared context (the topic, the words used) and leaves behind a "vector" of the specific linguistic phenomenon being tested. This allows us to see that even when a model's output is flawed, its internal representations may already have captured the underlying rules of the language.

Papers in this version

Showing 60 of 460 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 214 papers
  2. Version 1 · 2026-10-04 · 460 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_minimal_pair_evaluation_targeted_syntact_df51c481,
  title = {minimal pair evaluation (targeted syntactic evaluation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimal-pair-evaluation-targeted-syntactic-evaluation/rdr_df51c481f5814b8481adb3516e71c4bf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_probing_language_models_with_linguistic__df51c481_v1,
  title = {Probing Language Models with Linguistic Minimal Pairs},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probing-language-models-with-linguistic-minimal-pairs/rdr_df51c481f5814b8481adb3516e71c4bf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}