The Dissociation Between Perplexity and Syntactic Competence

40 concepts26 papers1 proposal

When evaluating language models, the default yardstick has long been perplexity—a measure of how well a model predicts the next word in a sequence. Intuitively, it seems reasonable that a model with lower perplexity must possess a sharper grasp of language, including its grammar.

However, a growing body of research reveals a fundamental flaw in this assumption: a model's predictive fluency frequently dissociates from its actual syntactic competence. A model can achieve state-of-the-art perplexity while remaining fundamentally blind to the hierarchical rules of grammar, while another model might exhibit higher perplexity yet possess a far more robust internal representation of structural syntax.

To understand why this dissociation occurs, we must look at how perplexity is calculated and where its blind spots lie.

The Illusion of the Average Metric

Perplexity is an aggregate metric. It computes the geometric mean of the probabilities assigned to every single token in a corpus. In natural language, however, tokens are not distributed evenly in terms of their syntactic importance.

As Hill et al. (2016) point out, standard perplexity averages performance across all word types. This means the metric is heavily dominated by high-frequency function words ("the", "is", "of") and local, highly predictable statistical collocations. If a model excels at guessing these local patterns, its perplexity will plummet.

Consequently, as Hu et al. (2020) demonstrate, minimizing perplexity does not mean a model has internalized the formal rules required for complex syntactic generalization. The aggregate nature of the metric masks critical deficiencies in grammatical understanding. A model can guess the next word accurately most of the time by relying purely on surface-level statistical heuristics, without ever learning the underlying structural hierarchies of the language.

Why Models Can "Cheat" Perplexity

Natural language is inherently hierarchical, structured around nested phrases rather than simple linear sequences. Yet, sequential language models optimized solely for next-token prediction often find a shortcut: they exploit linear regularities instead of learning hierarchical rules.

Consider a classic subject-verb agreement problem where a long relative clause separates the subject from its verb:

The key to the complex cabinets that the janitors opened is on the table.

To predict the singular verb is instead of the plural are, a model must understand that key is the true structural subject, ignoring the linearly closer plural nouns cabinets and janitors.

Because these long-distance dependencies are relatively rare in training corpora, a model's failure to resolve them correctly has a negligible impact on its overall perplexity. Yedetore et al. (2023) evaluated models trained on child-directed speech and found that even when models achieved low perplexity, they consistently favored incorrect linear rules over hierarchical ones when executing grammatical transformations.

Similarly, Marvin et al. (2018) showed that while standard sequential models excel at simple, local grammar, they struggle significantly with these long-distance dependencies. The models are highly effective at general word prediction because they memorize common word associations, but they fail to capture the underlying structural constraints of the language.

The Empirical Decoupling of Fluency and Structure

This mismatch is not just theoretical; it has been repeatedly demonstrated through systematic empirical testing. When researchers track both perplexity and targeted syntactic performance across different architectures and data scales, the two metrics frequently diverge.

In a comprehensive study of multiple model-data combinations, Hu et al. (2020) confirmed a clear dissociation: a model’s internal architecture and inductive bias are often far more critical for mastering grammar than simply scaling up data to lower perplexity. This finding is reinforced by Pérez-Mayos et al. (2021), who discovered that models with lower perplexity do not necessarily achieve superior scores on targeted syntactic evaluations, proving that next-token accuracy is an unreliable indicator of structural language understanding.

  Standard Optimization Objective
  [ Next-Token Prediction ] ---> Minimizes Perplexity ---> High Surface Fluency
                                                                |
                                                           (Dissociation)
                                                                v
  Explicit Structural Bias  ---> Masters Hierarchical ---> Robust Syntactic
  [ e.g., Tree Architectures ]    Grammar Constraints      Competence

The dissociation becomes even more striking when evaluating syntax-aware architectures. Models that explicitly incorporate tree structures—such as those explored by Zhao et al. (2025)—frequently show vastly improved performance on complex syntactic tasks despite exhibiting higher raw text perplexity scores.

Because these models are constrained to process text through structural hierarchies, they lose some of the fluid, statistical flexibility that standard Transformers use to minimize perplexity on arbitrary text. Yet, their structural intelligence is demonstrably superior. As Enguehard et al. (2017) demonstrated, forcing a model to learn specific grammatical objectives improves the internal consistency of its syntactic representations without necessarily lowering its overall perplexity.

Designing Better Probes for Syntactic Intelligence

If perplexity cannot be trusted to measure grammatical mastery, how can we accurately evaluate a model's linguistic competence? The research community has shifted toward targeted behavioral testing and structural probing.

One common approach is the use of minimal pairs, introduced at scale by Marvin et al. (2018). This method compares the model's probability assignments for two sentences that are identical except for a single grammatical feature (e.g., "the boy eats" vs. "the boy eat"). If the model assigns a higher probability to the grammatical sentence, it passes.

However, as Davis et al. (2022) argue, binary acceptability judgments set a low bar. They only confirm a statistical preference between two options rather than verifying if a model truly understands the underlying rule. By using Grammatical Error Detection (GED) as a more rigorous probe—requiring models to pinpoint the exact location and nature of a violation within a single string—the authors revealed that a model's syntactic knowledge is often surprisingly brittle and dependent on surface patterns.

This limitation is also central to the CheckList methodology proposed by Ribeiro et al. (2021). By breaking down evaluation into specific linguistic capabilities and behavioral sanity checks, they demonstrated that state-of-the-art models with excellent benchmark accuracy regularly fail simple structural tests involving negation or basic logic.

Finally, to isolate structural generalization from pure memorization, Patil et al. (2024) introduced Filtered Corpus Training (FiCT). By systematically removing specific grammatical constructions from the training data, they showed that a model's predictive fluency (perplexity) fails to predict its ability to generalize linguistic rules from indirect evidence.

Ultimately, these diverse methodologies converge on a singular conclusion: mastering the statistical regularities of text generation is fundamentally distinct from mastering the structural rules of language. To build models that truly understand syntax, we must optimize and evaluate them for structural competence, rather than chasing the deceptive proxy of lower perplexity.

Key Foundational Work

Exploring the Syntactic Abilities of RNNs with Multi-task Learning (2017) — contributed 1 concept elements to this direction.
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations Felix Hill (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_perplexity_syntax_dissociation_1d10f84a,
  title = {perplexity-syntax dissociation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perplexity-syntax-dissociation/rdr_1d10f84af6414c37afc9fd21b095019c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_perplexity_syntax_dissociation_1d10f84a_v1,
  title = {perplexity-syntax dissociation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perplexity-syntax-dissociation/rdr_1d10f84af6414c37afc9fd21b095019c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake