Measuring the Curvature of Syntax in Language Models with Controlled Grammars

Two recent results appear to support the same claim, that language models store sentence structure in a curved, tree-like geometry. Probes that read parse trees out of a model's hidden states work better when distances are measured in hyperbolic space. Hyperbolic space is a geometry in which room grows exponentially with distance from the center, so trees fit without their leaves crowding together. The evidence has a gap, though. It was collected either on natural treebanks, where tree distance and word-order distance are heavily entangled, or on reasoning chains, which are not trees at all. Separately, grammar-generated test sentences have shown that the standard flat (Euclidean) probe mostly tracks how far apart words sit in the sentence rather than their syntactic relation. No one has checked whether the hyperbolic advantage holds once that confound is removed.

We propose to generate sentences from a probabilistic grammar that sets three quantities independently: branching factor, nesting depth, and linear distance. We then fit a structural probe whose curvature is a learned parameter. At zero curvature it reduces to an ordinary Euclidean distance probe. The curvature the probe settles on becomes a measurement of the model's representation, rather than an assumption built into the probe. We ask three questions. Does that fitted curvature grow with the tree's branching factor and depth, as hyperbolic theory predicts? Does it survive controls that rule out a merely more flexible probe? Does it predict where models break when they face deeper recursion than they were trained on?

Core Claim

If a model encodes syntax in a hyperbolic-like way, a probe with learnable curvature fitted to its hidden states should prefer larger curvature exactly when the stimulus trees branch more and nest deeper. The reason is geometric. A tree with branching factor $b$ has about $b^d$ nodes at depth $d$. A flat space of fixed dimension has room for only polynomially many of them before they crowd, so the flat probe's distortion should grow with $b$ and $d$ at fixed probe rank, while a hyperbolic probe's should not.

The fitted curvature should not move with linear distance once tree distance is held fixed, and it should not appear for untrained models or for position-only inputs. If the curvature instead tracks linear distance, or appears equally in the controls, then the published hyperbolic advantage reflects probe flexibility or leftover word-order confounds, not the model's geometry. Either outcome is informative. Only a grammar in which these factors are decorrelated can tell them apart.

Relation to Prior Work

Two papers motivate the question.

Hyperbolic Embedding for Finding Syntax in BERT maps BERT states into the Poincaré ball, a model of hyperbolic space drawn as the unit disk, and fits unsquared hyperbolic distance to dependency-tree distance on English Universal Dependencies. It reports a UUAS (the share of tree edges recovered correctly) of 78.51% and a DSpr (rank correlation between probe distances and true tree distances) of 83.96%. It uses only natural text, and it fixes the curvature in advance.

Hyperbolic Geometry of Reasoning: Probing LLM Hidden States finds that on PrOntoQA a hyperbolic probe holds a Spearman correlation of about 0.967 in DeepSeek-R1's final layer, where the Euclidean probe falls to about 0.488. PrOntoQA reasoning chains are linear, however. On a chain, tree distance is just the difference in position. Any exponential-capacity advantage should be irrelevant there, so that result leaves open why the hyperbolic probe helps.

Two further papers constrain the design.

Probing Syntax in Large Language Models: Successes and Remaining Challenges builds controlled prepositional-phrase, center-embedding and right-branching stimuli. It shows that Euclidean probe accuracy drops "a linear distance of just 3 words" out, regardless of the true syntax. That paper tested only Euclidean and polar probes. Its stimulus logic is the template we extend to control branching as well.

A Non-Linear Structural Probe shows that a more flexible Euclidean probe using an RBF kernel (a similarity that decays smoothly with distance) raises Tamil UUAS by 8.44 points but hurts DSpr, because its distances are bounded. This is a warning that probe gains can come from flexibility and not from matching the right geometry.

The method comes from A Structural Probe for Finding Syntax in Word Representations, which supplies the linear projection and distance-matching objective, and from Poincaré Embeddings for Learning Hierarchical Representations, which supplies the Poincaré geometry.

The grammar side draws on COGS, a benchmark generated from a probabilistic grammar in which standard sequence models score near zero once nesting exceeds the training depth. It also draws on Learning Syntax Without Planting Trees, which shows that competing linear and hierarchical solutions coexist inside small Transformers trained on ambiguous grammars. Both papers provide trained-from-scratch settings where we control exactly which depths a model has seen.

Research Program

The stimulus grammar. We will write a probabilistic context-free grammar over a BLiMP-style English lexicon. Its rules for prepositional-phrase attachment, center embedding, right branching, and multi-argument coordination let us set the branching factor $b \in \{1,2,3,4\}$ and nesting depth $d \in \{1,\dots,6\}$ separately. By choosing which modifiers fill the intervening slots, we also set the linear distance between any two words at a fixed tree distance.

The output is a fully crossed design. Some cells hold word pairs that are close in the tree but far apart in the sentence. Other cells hold pairs that are far apart in the tree but adjacent in the sentence. We add the Probing-Syntax prepositional-phrase, center-embedding and right-branching stimuli as they are, to allow a direct replication.

The probe. A learned projection $B$ maps each hidden state $h_i$ to $z_i = \exp_0^c(B h_i)$. Here $\exp_0^c$ is the standard map that sends a flat vector into the Poincaré ball of curvature $-c$. We then minimize

$$\sum_{i,j} \left| d_c(z_i, z_j) - d_T(w_i, w_j) \right|,$$

where $d_T$ is the true tree distance between words $w_i$ and $w_j$, and $d_c$ is the hyperbolic distance at curvature $c \ge 0$. The curvature $c$ is learned jointly with $B$. As $c \to 0$, $d_c$ becomes twice the ordinary unsquared Euclidean distance, so the flat probe is a special case and not a separate model.

A companion depth probe reads each word's tree depth from the norm $\|z_i\|$. We report the fitted curvature $\hat c$ with bootstrap intervals across seeds. We break results down by $b$, $d$, linear distance, layer, and probe rank $k \in \{2,4,8,16,64\}$.

The models. We will probe BERT-Large for comparability with the treebank results, GPT-2 for comparability with BLiMP, and Qwen2.5-7B for comparability with the reasoning-geometry paper. We will also train small Transformers from scratch on the grammar, using sentences of depth at most 2 only, following the COGS and Planting-Trees protocols. The small models let us relate geometry to behavior on unseen depths.

Baselines

The naive reference points are a linear-chain predictor, which guesses that each word attaches to its neighbor, and position-only inputs. The linear chain scores 48.9 UUAS on the Penn Treebank in the structural-probe paper. For position-only inputs, the same curvature probe is fitted to sinusoidal position vectors and to an untrained copy of each model. If the curved probe recovers deep trees from these inputs, its gains on trained models cannot be credited to the models.

The strongest existing method is the squared-distance Euclidean structural probe from the structural-probe paper. It reports 82.5 UUAS and 0.87 DSpr on BERT-Large, layer 15, on the Penn Treebank. Also in this tier is the fixed-curvature Poincaré probe from the hyperbolic BERT paper, which reports 78.51% UUAS and 83.96% DSpr on UD. For the controlled stimuli, we found no reported absolute number on Lacuna for either Euclidean probes on the prepositional-phrase, center-embedding and right-branching data or hyperbolic probes on any controlled grammar. We will therefore reproduce both on our splits first.

The RBF-kernel probe from the non-linear probe paper is the strongest flexibility-matched competitor. It is included because it separates "more expressive" from "curved."

All probes get the same stimuli, layers, rank grid, optimizer, 50-trial tuning budget, and five seeds. Parameter counts are matched at each rank.

The key ablation compares three probes: the learnable-curvature probe, the same probe with $c$ frozen at 0 (the unsquared Euclidean probe), and the squared Euclidean probe. The gap between the squared and unsquared Euclidean probes measures what the choice of loss contributes. The gap between the unsquared Euclidean and learned-curvature probes measures what curvature contributes.

The proposal beats these baselines under two conditions. First, the learned-curvature probe must reduce distortion (mean absolute distance error) relative to the $c=0$ probe in high-branching, deep cells, with the gap widening as $b$ and $d$ grow at low rank. Second, it must show no such gain on position-only and untrained controls.

Why not just…?

Why not just raise the rank of the Euclidean probe? The structural-probe paper found syntax lives in a compact subspace, with little gain beyond 64 to 128 dimensions. However, that was on natural sentences, where trees are shallow and narrow. The geometric argument predicts that the rank a flat probe needs grows with $b^d$. We measure this directly by sweeping rank in every $(b,d)$ cell. If a rank-64 Euclidean probe matches the curved probe everywhere in our grammar, that is a real negative result: curvature would buy only compression, not recoverability, and we will report it as such.

Why not just use an unsquared Euclidean distance? This is the most serious threat. Part of the published hyperbolic advantage may come from dropping the square, not from the curvature. That is why our probe contains the unsquared Euclidean probe as its $c=0$ case. If $\hat c$ stays near zero, or the $c=0$ probe matches the curved probe cell for cell, the hyperbolic story reduces to a change of loss.

Why not just use an RBF or other non-linear Euclidean kernel? The non-linear probe paper already shows RBF kernels improve local attachment but degrade distance correlation, because their distances are capped. Our grammar makes that cap testable. At tree distances beyond about 6, only unbounded metrics can keep their rank correlation. We report DSpr stratified by tree distance to show where each probe saturates.

Why not just test behavior, such as agreement accuracy, instead of probing geometry? Behavioral tests like BLiMP, where GPT-2 scores 83.0% against 88.6% for humans, say whether a model succeeds, not how it represents structure. We keep behavior as the outcome to be explained. For each item we ask whether probe distortion predicts the model's agreement error at that item's depth.

Proposed Experiments and Decisive Checks

Replication. We first reproduce the linear-distance collapse of Euclidean probes on the Probing-Syntax stimuli, and then test whether the learned-curvature probe shows the same collapse. We measure UUAS and DSpr as partial correlations that control for linear distance.

Factorial study. Across the $(b, d, \text{linear distance})$ grid, the decisive pattern is threefold. The fitted curvature $\hat c$ should rise with $b$ and $d$. It should stay flat along linear distance. It should be near zero for position-only and untrained controls.

Chain versus tree. We repeat the analysis on right-branching-only sentences, which are chains, and compare against branching sentences. If curvature helps equally on chains, it would suggest that the PrOntoQA result reflects something other than hierarchy.

Recursion wall. For the from-scratch Transformers trained on depth at most 2, we fit probes on depths up to 2 and evaluate them on depths 3 to 6. The check is whether a depth-norm and curvature fit that extrapolates predicts which seeds generalize, given the seed sensitivity reported in Planting-Trees. We relate this to exact-match accuracy on the COGS recursion splits, where standard sequence models score near zero and out-of-distribution accuracy is 16–35%. A positive correlation across seeds would tie the geometry to generalization. A null result would show that probe geometry and generalization come apart.