Do Speech Models Build Dependency Trees? A Context-Controlled Structural Probe on Spoken UD

Self-supervised speech models learn from raw audio alone and never see text. Over a million hours of listening, do they come to represent who modifies whom in a sentence, that is, the subject of a verb, the noun a preposition attaches to, the clause an adverb belongs to? Existing evidence says something grammatical is in there, but only at the level of whole sentences. A linear readout can predict how deep a sentence's parse tree is, and a classifier can tell grammatical from ungrammatical audio. Neither shows whether the model's internal geometry encodes the word-to-word tree that linguists annotate. We propose to test this directly. We render Universal Dependencies (UD) treebanks as speech, pool the frozen speech model's frame vectors into one vector per word, and ask whether a small linear map recovers the full directed, labeled dependency tree.

The part that makes this measurement trustworthy is a control. Speech vectors for a word carry a great deal of acoustic and lexical identity, and a probe could reconstruct plausible trees from "this is the word the" plus word position, with no real syntax involved. We therefore compare each word's in-sentence vector against the vector the same model produces when it hears only that word's audio segment, cut out of the same recording. The two inputs have identical acoustics and coarticulation, and only the second lacks sentence context. The gap between them, measured layer by layer and language by language, estimates how much tree structure speech models compute from context. That quantity has not been measured.

Core Claim

Our hypothesis is that frozen self-supervised speech encoders contain a linearly recoverable dependency geometry over words, with three properties. It exceeds what word identity and position alone allow. It peaks in middle layers and then collapses, and the collapse is steeper for tree distances than for relation labels. In cross-lingual encoders it is partly shared across languages. The deliverable is a reusable spoken-UD probing benchmark with the context control built in. Speech benchmarks currently lack such a test: neither SUPERB nor SUPERB-SG includes a syntactic task.

The method comes from text. A Structural Probe for Finding Syntax in Word Representations learns a matrix $B$ so that the squared distance $\lVert B(h_i - h_j)\rVert^2$ between the transformed vectors $h_i, h_j$ of words $i$ and $j$ approximates the number of tree edges between them. Probing for Labeled Dependency Trees, also called DEPPROBE, adds a second linear map $L$ that classifies each word's relation to its head. It then decodes directed, labeled UD trees with about 127k parameters. We reuse both unchanged. The new parts are the input modality, the context control, and the analyses they make possible.

Two speech papers motivate the proposal and also constrain it. Wave to Syntax found that syntax in wav2vec 2.0 and HuBERT peaks mid-network and dips at the top. It probed only English sentence-level constituency depth, and it warned that the signal is entangled with lexical identity and duration. That warning is the reason our design centers on a control. Layer-wise Minimal Pair Probing showed that HuBERT exceeds 90% on synthesized BLiMP acceptability pairs and that ASR-trained encoders keep syntax into their final layers. Its probes judge whole utterances, so they cannot say which attachment a model gets right or wrong. Our probe can.

Research Program

Data. UD sentences are text, so we build the audio. For each of roughly ten UD treebanks spanning several families, we synthesize the training and test sentences with an off-the-shelf multilingual text-to-speech system. We choose treebanks with usable synthesis, including English EWT and a selection of Romance, Slavic, Indic, Turkic and East Asian treebanks. The synthesizer's duration outputs give exact word boundaries. The minimal-pair study already used synthesized audio, so our main results share its main weakness: synthesized speech is cleaner and more regular than natural speech. To address this, we will have human speakers read a few hundred test sentences in English and one other language. We will align those recordings to the text, and every headline result will be reported on this natural subset as well.

Word boundaries are an oracle input. This study tests geometry, not word segmentation, and we say so explicitly.

Representations. For each layer $\ell$ of wav2vec 2.0, HuBERT, XLS-R (XLS-R) and an ASR-trained encoder (the Whisper encoder, as in the minimal-pair study), we mean-pool frames within each word to get a contextual vector. We then extract that word's audio span, add a few tens of milliseconds of padding, run the span through the same frozen model alone, and pool again to get a cropped vector. The cropped vector has the same sound and the same coarticulation, but the model has heard nothing from the rest of the sentence. The quantity we report is the context gain

$$\Delta_\ell = \mathrm{UUAS}_\ell(\text{contextual}) - \mathrm{UUAS}_\ell(\text{cropped}),$$

where UUAS is the fraction of gold undirected edges recovered by the minimum spanning tree over predicted distances. We report the same difference for labeled attachment score (LAS) and relation accuracy.

Analyses. The first analysis takes up an open question from the file: when the top-layer dip occurs, do tree distance ($B$) and relation labels ($L$) degrade together? We plot both curves per model. This also tests the claim in Iterative Refinement, Not Training Objective that HuBERT's repeated re-clustering improves word-level information in its upper layers. Their claim predicts a smaller dip for HuBERT than for wav2vec 2.0 at equal size. If public checkpoints from different HuBERT iterations are available, we compare them directly.

The second analysis is cross-lingual. We train $B$ on XLS-R vectors for one language, apply it zero-shot to the others, and correlate the resulting transfer matrix with the text-side transfer matrix from On Language Spaces, Scales and Cross-Lingual Transfer of UD Parsers. That paper found that its text attachment probe predicts parser transfer with Spearman $\rho = -0.66$. If speech syntax shares a structure across languages, the speech transfer pattern should resemble the text one. If the speech pattern instead follows acoustic or phonological similarity between languages, the shared signal is not syntactic.

The third analysis is a single targeted check. We convert the same trees to SUD (SUD), an annotation in which function words head their phrases. We ask which scheme's distances the speech geometry fits better. Because the round-trip conversion is near-lossless (median LAS 94.1%), any difference in fit reflects the headedness choice rather than information lost in conversion.

Baselines

The naive reference points are structural, and they require no model. The first is the linear-chain baseline, which connects each word to its neighbor. On the Penn Treebank, A Structural Probe for Finding Syntax in Word Representations reports a UUAS of 48.9 for it. We will recompute it on every UD test set. The second applies the same probe to word position, onset time and duration features. This captures everything a probe could learn from timing alone. A speech model's contextual UUAS must clear both baselines on the same test sentences before we interpret anything.

The acoustic floor is the identical probe trained on pooled log Mel-filterbank features. SUPERB-SG reports that these features remain competitive on generative tasks, so they are a fair test of whether raw spectra already carry the needed signal. They receive the same pooling, probe rank and tuning budget as the learned models.

The key ablation is the cropped-segment control described above. It is the baseline the proposal exists to beat. A model "has syntax" in our sense only if $\Delta_\ell$ is reliably positive, with a bootstrap confidence interval over sentences that excludes zero. For calibration we compare against text. ELMo's non-contextual layer reaches a UUAS of 77.0 on the Penn Treebank, a strong lexical-only reference. That is why lexical controls matter so much here.

The strongest existing method is DEPPROBE on frozen mBERT text vectors for the same sentences. Its paper reports a mean LAS of 60 across 13 UD v2.8 languages. We rerun it on our exact treebank splits so the comparison is like-for-like. The speech models will almost certainly fall short of it. Beating it is not the point. It sets the scale: we report speech context gain as a fraction of mBERT's context gain, measured as mBERT's best layer minus its embedding layer.

As an engineering upper reference, we run a cascade on the same audio: ASR transcription followed by a full text UD parser. No reported result was found on Lacuna for cascaded parsing of synthesized or read UD speech. The closest figure is LAS above 87% for Stanza on manual adult-speech transcripts, reported in Dependency Parsing Evaluation for Low-resource Spontaneous Speech. No reported result was found on Lacuna for any structural or labeled dependency probe on speech representations.

All probes share one protocol. We tune the probe rank, chosen among 64 and 128, and the learning rate on the English EWT development set, as DEPPROBE does. Every representation receives the same number of tuning trials, and test sets are touched once.

Why not just…?

Why not just transcribe with ASR and parse the text? This would give better trees, and we expect it to win on LAS. But it answers a different question: whether syntax is recoverable from audio by any route, rather than whether the speech encoder represents it. A text parser will also never use information that lives only in the audio. To make the contrast informative, we report the cascade alongside the probe. We also run the probe on a prosody-flattened resynthesis of the same audio. If contextual UUAS drops when pitch and pause cues are removed, the speech geometry carries attachment information that a transcript discards. If it does not drop, the cascade critique largely stands for read speech, and we will say so.

Why not just use the existing sentence-level probes? Tree-depth regression and minimal-pair accuracy already show "syntax is present." They collapse an entire tree into one number, so they cannot separate tree distances from relation labels, locate failures by relation type, or support cross-lingual transfer analysis by attachment. A word-level tree probe can do all three. The cost is that we need word boundaries, which the sentence-level probes avoid.

Why not just use a more expressive probe, such as an MLP or a biaffine parser on frozen features? A high-capacity probe can compute syntax itself from lexical identity, which is exactly the confound we are trying to remove. DEPPROBE's $\rho = 0.97$ agreement with a 183M-parameter parser on text suggests linear capacity is sufficient when the geometry exists. As a secondary check, we train a small MLP on both the contextual and cropped vectors. If the MLP lifts both by similar amounts, $\Delta_\ell$ remains the meaningful quantity. If the MLP closes the gap only on contextual vectors, that is evidence of syntax stored nonlinearly, and we will report it rather than hide it.

Why not just control with words spoken in isolation, or with text embeddings? Words spoken in isolation differ acoustically from the same words in running speech, so a gap could reflect a change in the audio rather than a change in context. The cropped segment keeps the audio fixed and removes only context. A text embedding control would come from a different encoder entirely, so it cannot isolate what this speech model adds by hearing the sentence.

Proposed Experiments and Decisive Checks

The first decisive experiment is the layer-by-layer context gain on English EWT, natural recordings included, for the four encoders. The hypothesis fails if every speech model's best-layer $\Delta_\ell$ is indistinguishable from zero. It also fails if contextual UUAS does not beat the position-and-duration probe. In either case, the earlier sentence-level findings would be read as lexical and timing effects.

The second is the comparison between the tree-distance and relation-label curves. We predict that the relation-label curve stays high for at least two layers past the point where the tree-distance curve drops in self-supervised models, and that neither drops for the ASR encoder. Identical curves would refute this prediction.

The third is the XLS-R cross-lingual transfer matrix. A positive Spearman correlation with the text transfer pattern, holding after we control for language-identification similarity in early XLS-R layers, would support a shared syntactic subspace. A correlation explained only by early-layer acoustic similarity would refute it.

The fourth is the gap between natural and synthesized speech. If contextual gain on human recordings is less than half the gain on synthesized audio, the synthesized benchmark overstates spoken syntax. We would then reframe the benchmark around natural speech and extend it toward spoken treebanks such as the Ukrainian parliamentary UD treebank (Developing a Universal Dependencies Treebank for Ukrainian Parliamentary Speech), wherever aligned audio can be obtained.