Recovering Syntax from the Geometry of Language Model Embeddings

This direction is split into other directions: structural probing. Its last version stays readable and citable here.

Version 1 of 131 papers31 conceptsfrozen 2026-10-04

When a Transformer processes a sentence, it represents each word as a high-dimensional vector. While these vectors are primarily optimized to predict the next token, researchers have discovered they contain a hidden, highly organized geometric structure that mirrors formal linguistics. By applying "structural probes," we can reveal that models like BERT and GPT-2 aren't just memorizing word patterns—they are constructing internal "maps" of grammar.

The Geometric Hypothesis of Syntax

In linguistics, the relationship between words is often represented as a tree. In a dependency tree, the "distance" between two words is the number of edges separating them in the hierarchy. For example, in the sentence "The chef who won the award cooked the meal," the words chef and cooked are syntactically close (subject and verb), even though they are separated by a five-word relative clause.

The core insight of structural probing, pioneered by John Hewitt and Christopher D Manning, is that if a model "understands" syntax, its internal vector space should be isomorphic to these tree structures. Specifically, the distance between two word embeddings in the model's hidden state should correlate with their distance in a linguistic parse tree.

Mapping Embeddings to Trees

To test this, researchers use a linear transformation—a simple matrix—to project the high-dimensional embeddings into a lower-dimensional "syntactic subspace."

The mechanism works as follows:

  1. Projection: A learned matrix $B$ transforms an embedding $h$ into a new vector $Bh$.
  2. Distance Metric: We measure the squared Euclidean distance between these projected vectors: $d(h_i, h_j) = \|B(h_i - h_j)\|^2$.
  3. Alignment: The probe is trained to find a matrix $B$ such that this distance $d$ matches the actual distance between words in a gold-standard dependency tree.

As Max Müller-Eberstein and colleagues demonstrate with their DEPPROBE method, the fact that a simple linear transformation can recover these trees suggests that syntax is a fundamental, linearly accessible property of the embedding space. If the information were buried in a non-linear way, a simple matrix wouldn't be able to "unfold" it so accurately.

Universal and Abstract Representations

This structural encoding isn't limited to English or even to meaningful text. A. Ethan and Ethan A. Chi found that Multilingual BERT (mBERT) develops a shared "syntactic subspace" across dozens of languages. This means the model uses the same geometric directions to represent "subject-verb" relationships in both English and Hindi, despite their different word orders.

Furthermore, David Arps showed that these probes work even on "nonce" sentences—grammatically correct but semantically nonsensical strings like "The colorless green ideas sleep furiously." This proves that the model's internal geometry is capturing pure structural relationships, independent of semantic cues or word associations. This capability even extends beyond natural language: Anja Reusch demonstrated that models trained on mathematics encode the hierarchical "Operator Trees" of formulas in a similar geometric fashion.

How Syntax Emerges Layer by Layer

Structural probing also allows us to "watch" a model build a sentence as data flows through its layers. Taiga Someya and the team introduced Derivational Probing to track this evolution. They found that different architectures have different "construction styles":

  • BERT tends to build structure from the bottom up, identifying local phrases in early layers and global sentence structure in later layers.
  • GPT-2 appears to develop local and global structures in parallel, likely due to its unidirectional objective.

This layer-wise analysis, also explored by Laura Pérez-Mayos, reveals that while models need significant data to refine these structures, the "syntactic backbone" of the model often stabilizes long before the model finishes its pre-training.

The Limits of Geometric Probing

While structural probes are powerful, they are not perfect mirrors of "understanding." Emmanuel Chemla and Yair Lakretz have cautioned that these probes can sometimes be "fooled" by superficial heuristics. Because words that are close in a tree are often close in a linear sentence, a probe might achieve high accuracy by simply relying on word proximity rather than true hierarchical depth.

Despite these challenges, the success of structural probing across languages, domains, and architectures suggests that the Transformer's success is deeply tied to its ability to organize information into a geometric proxy for human grammar. By identifying these "latent structural dimensions," researchers like Anran Hao are now moving from observation to intervention—using these subspaces to "reinforce" syntactic knowledge and improve performance on complex tasks like event extraction.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_recovering_syntax_from_the_geometry_of_l_76f0eed7,
  title = {Recovering Syntax from the Geometry of Language Model Embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recovering-syntax-from-the-geometry-of-language-model-embeddings/rdr_76f0eed78a854e8d8de3a49bc42d74d5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_recovering_syntax_from_the_geometry_of_l_76f0eed7_v1,
  title = {Recovering Syntax from the Geometry of Language Model Embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recovering-syntax-from-the-geometry-of-language-model-embeddings/rdr_76f0eed78a854e8d8de3a49bc42d74d5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}