Probing Language Models with Structural Perturbations

102 concepts87 papers2 proposals

When a large language model (LLM) correctly answers a complex question, solves a logic puzzle, or describes an image, it is tempting to assume it possesses a human-like grasp of grammar, syntax, and compositional logic. However, modern neural architectures are notorious for exploiting statistical shortcuts—predicting labels based on word co-occurrences and surface-level patterns rather than deep structural understanding.

To determine whether a model truly understands the hierarchical rules of language or is simply acting as a sophisticated pattern matcher, researchers use a diagnostic paradigm known as structural perturbation analysis (or input permutation testing). The core intuition is simple: if you scramble the words of a prompt into a "word salad" and the model's performance remains completely unchanged, the model never relied on the syntax to begin with; it was treating the input as an unordered "bag of words." By systematically breaking the structure of text at different granularities and measuring the resulting performance degradation, we can map out exactly what structural elements a model relies on.

Original:   "The chef chopped the crisp onion."  -> Model predicts: Success
Perturbed:  "Onion crisp the chopped chef the."  -> Model predicts: Success (Shortcut exposed!)

The Granularities of Disruption

Structural perturbation is not just about random shuffling; it is a precision tool. By controlling how and where text is disrupted, researchers can isolate specific linguistic properties.

Word and n-gram Shuffling

The most direct way to test a model's reliance on linear sequence is to reorder words or short phrases. Anh Totti Nguyen et al. evaluated state-of-the-art Natural Language Understanding (NLU) models by randomly reordering n-grams of varying sizes. They discovered that models like BERT and RoBERTa often maintain high accuracy even under severe word-order disruption, suggesting that many standard benchmarks fail to test true compositionality. Similarly, Chinnadhurai Sankar et al. applied word-level syntactic perturbations to conversation histories in dialog systems, demonstrating that both recurrent and transformer-based models are surprisingly indifferent to the sequential structure of a conversation.

Boundary-Sensitive Swaps

To look beyond linear order and probe for hierarchical phrase structures, researchers can target syntactic boundaries. Matteo Alleman et al. introduced interventional perturbations that compare the impact of swapping adjacent tokens within the same syntactic unit versus tokens that cross a structural boundary. If a model's internal representations are more heavily distorted by boundary-crossing swaps, it proves that the model's geometry is sensitive to hierarchical phrase organization.

Character-Level Perturbations

Perturbations can also be applied inside words to stress-test sub-word tokenizers and morphological understanding. Hitomi Yanaka et al. and the authors of A Cross-linguistic Analysis of the Effects of Character-level Information in Neural Models systematically shuffled character sequences within words. They found that if a neural parser performs equally well when character order is randomized, the architecture is merely recognizing the presence of a character set rather than capturing genuine morphological structure.


Unmasking Spurious Shortcuts

Structural perturbation analysis has served as a sobering reality check across several domains, exposing instances where models achieve high benchmark scores via superficial heuristics.

Logical and Numerical Reasoning

In reading comprehension and logic tasks, models often bypass actual reasoning by matching keywords. Hadeel Al-Negheimish et al. scrambled input text on the DROP benchmark, revealing that top-performing models could still answer numerical questions correctly without their structural context. In the realm of logical deduction, Houman Mehrafarin et al. randomized the word order within supporting premises for transitive reasoning tasks ($A \to B, B \to C \implies A \to C$). The models' resilience to this scrambling unmasked a reliance on named entity recognition and keyword matching rather than genuine logical deduction.

Vision-Language Compositionality

Vision-Language Models (VLMs) are particularly susceptible to bag-of-words shortcuts, often matching images to captions based solely on isolated objects while ignoring how they interact. To expose this, Tristan Thrush created the Winoground benchmark, which uses pairs of captions containing identical words but different meanings based entirely on syntax (e.g., "some plants surrounding a lightbulb" vs. "a lightbulb surrounding some plants"). By holding vocabulary constant, this perturbation forces models to rely on structural understanding, revealing that state-of-the-art VLMs frequently perform no better than random chance.

[Image A: Plants around a bulb]   [Image B: A bulb around plants]
Caption 1: "some plants surrounding a lightbulb"
Caption 2: "a lightbulb surrounding some plants"

Similarly, Sonal Kumar et al. demonstrated that reversing the sequence of compound nouns (e.g., "peanut butter") causes severe classification drops in CLIP, while Lin Ju utilized structural permutations in the Ref-Adv benchmark to expose how multimodal models bypass multi-step visual reasoning in referring expression tasks.


From Diagnosis to Structural Awareness

Understanding why and when models fail under structural perturbation has allowed researchers to move from purely diagnosing weaknesses to actively mitigating them.

Forcing Syntactic Awareness during Training

If models exploit shortcuts because standard datasets allow them to, one solution is to change the training objective. Hadeel Al-Negheimish et al. introduced Forced Invalidation (FI), a strategy where language models are explicitly trained to categorize permuted or "word salad" sequences as invalid. This negative constraint prevents the model from relying on simple word co-occurrences and forces it to encode grammatical structure.

Disrupt-and-Recover Frameworks

The sensitivity of language models to structural order can also be turned into a feature. The authors of D&R: Recovery-based AI-Generated Text Detection via a Single Black-box LLM Call developed an AI-text detection framework based on a disrupt-and-recover loop. By segmenting a text, randomly shuffling tokens within those chunks, and asking an LLM to "fix" the syntax, they discovered that machine-generated text is recovered with significantly higher fidelity than human writing. Because AI text aligns closely with the internal statistical priors of language models, its underlying structure is far easier for a model to reconstruct after a perturbation.

Model-Adaptive Prompting

Finally, understanding structural sensitivity helps optimize how we interact with models. Yuyan Chen et al. developed MAPO (Model-Adaptive Prompt Optimization), a framework that automatically rearranges word order and adjusts prompt syntax. While these structural adjustments do not alter the semantic meaning for human readers, they dramatically improve performance by tailoring instructions to the specific, underlying syntactic preferences and positional biases of individual LLMs.

Key Foundational Work

Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (2016) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 91 papers
  2. Version 1 · 2026-10-04 · 188 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_structural_perturbation_analysis_ddea5846,
  title = {structural perturbation analysis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-perturbation-analysis/rdr_ddea5846bb5a442383e949d44b1b0897},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_structural_perturbation_analysis_ddea5846_v2,
  title = {structural perturbation analysis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-perturbation-analysis/rdr_ddea5846bb5a442383e949d44b1b0897/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake