Modern language models often appear to possess a sophisticated grasp of human grammar, yet researchers frequently question whether this is true "understanding" or merely a high-dimensional exploitation of surface-level statistics. To resolve this, a growing body of research employs structural perturbation probing—the practice of systematically "breaking" the linguistic structure of inputs to observe how model performance or internal representations degrade. By comparing a model's reaction to natural language versus "impossible" or scrambled versions of it, we can diagnose whether a system relies on hierarchical rules or simple bag-of-words heuristics.
The Bag of Words Heuristic
The simplest way to test a model’s reliance on syntax is to destroy it. If a model performs just as well on a shuffled sequence of words as it does on a coherent sentence, it is likely treating the input as a "bag of words." Chinnadhurai Sankar and colleagues demonstrated this in neural dialog systems, finding that both recurrent and Transformer models were surprisingly indifferent to reversed or shuffled conversation histories (Do Neural Dialog Systems Use the Conversation History Effectively?).
This phenomenon extends to multi-modal tasks. In Vision-and-Language Navigation (VLN), Wang Bill Zhu found that pretraining performance remained stable even when instructions were replaced with "gibberish" or shuffled tokens, suggesting models were simply mapping keywords to visual landmarks rather than parsing the logic of the command (Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions?). Similarly, Avinash Madasu showed that video retrieval models often ignore word order entirely, behaving as keyword detectors rather than semantic processors (ICSVR: Investigating Compositional and Semantic Understanding in Video Retrieval Models).
Diagnosing Internal Hierarchy
While a model might act like a bag-of-words detector on a specific task, its internal layers may still encode structural information. Researchers use interventional perturbations to "peek" at these hidden states. Matteo Alleman and Yoon Kim used boundary-sensitive perturbations—swapping words that cross syntactic boundaries versus those within the same phrase—to show that BERT’s internal geometry is indeed sensitive to hierarchical organization (Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models).
This aligns with the "pipeline" view of Transformers proposed by Ganesh Jawahar and Benoît Sagot, where lower layers capture surface features and middle layers resolve syntax (What Does BERT Learn about the Structure of Language?). In fact, Jiaxi Li demonstrated that by measuring how much a model’s hidden states "distort" when a span of text is perturbed, one can extract highly accurate syntactic trees without any explicit training, proving that Masked Language Models are implicit parsers (Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers).
Hallucinations of Normality
A recurring finding in structural probing is that models are often too robust to noise. In Machine Translation, Koustuv Sinha and Adina Williams observed "hallucinations of normality": when given an ungrammatical or perturbed source sentence, models often ignore the actual input structure to produce a standard, grammatical translation (Sometimes We Want Ungrammatical Translations).
This suggests a "knowledge-action gap." A model might recognize that a sentence is weird (internal sensitivity), but its training objective—to produce high-probability natural language—forces it to smooth over those weirdnesses. Antonia Karamolegkou found a similar gap in ambiguity detection: while LLMs fail to identify linguistic ambiguity when asked directly, linear probes can extract that information from their internal layers with high accuracy (Trick or Neat: Adversarial Ambiguity and Language Model Evaluation).
Inductive Biases and Impossible Languages
If models were truly blank slates, they should be able to learn any statistical pattern, no matter how unnatural. However, Julie Kallini and Christopher Potts challenged this by training models on "impossible" synthetic languages—corpora where natural language rules were replaced by random shuffles or non-hierarchical patterns (Mission: Impossible Language Models). They found that models struggle significantly more with these unnatural structures, revealing a clear inductive bias toward the hierarchical and local dependencies found in human speech.
This bias is what makes structural perturbation such a powerful tool. By measuring the "DeltaScore"—the drop in a model's assigned probability when a text is perturbed—researchers like Zhuohan Xie can even evaluate the quality of generated stories. High-quality text has a fragile structural integrity that breaks when shuffled, whereas low-quality text is already so structurally weak that perturbations barely change its likelihood (DeltaScore: Evaluating Story Generation with Differentiating Perturbations).