The Linguistic Structure of Dependency-Conditioned Embeddings

Bach Phan-Tat, Tung Kieu, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk SpeelmanarXiv 2026 · arXiv

TL;DR

This paper explores the "hidden" geometry of dependency-conditioned word embeddings—representations that treat a word and its syntactic role (e.g., "bank" as a direct object) as a single unit. The authors discover that these embeddings are globally organized by the part-of-speech of their context, while locally, they act as powerful filters that separate a word's different meanings based on its grammatical usage.

What Problem Does This Paper Address?

Standard distributional semantic models (like Word2Vec or GloVe) typically produce a single vector for each lemma, effectively squashing all of a word's different meanings and syntactic functions into one point in space. While dependency-based models use syntax to inform these vectors, the actual lemma-relation pairs (e.g., "bank" acting as a compound noun vs. "bank" as an oblique object) are rarely studied as a standalone semantic space.

The authors set out to map this space. They want to know: How is the geometry of these syntactic units organized? Does the structure depend on the learning algorithm (count-based vs. predictive)? And can these representations naturally distinguish between different senses of polysemous words?

Methods

The researchers trained four different representation models on the WikiText-103 corpus:

  1. PPMI-5000: A sparse, count-based model using the 5,000 most frequent contexts.
  2. PPMI-SVD: A dense version of the count-based model reduced to 300 dimensions.
  3. Original SGNS: The context vectors from a standard dependency-based Skip-Gram model.
  4. Reversed SGNS: A Skip-Gram model specifically trained to treat lemma-relation pairs as the prediction targets.

To analyze the geometry, they used t-SNE for visualization and Hierarchical Agglomerative Clustering (HAC) to find groupings of relations. Crucially, they employed within-lemma centering—subtracting a word's average vector from its relation-specific vectors—to isolate the geometric "fingerprint" of the syntactic relation itself, independent of the word's identity.

Datasets

  • Training: WikiText-103 (approx. 103 million tokens from Wikipedia), parsed using the Stanza toolkit.
  • Evaluation: The WordSim-353 benchmark (Similarity and Relatedness subsets) was used to test how well these embeddings align with human semantic judgments across 14 different dependency relations.

Key Insights and Ideas

  • Context POS is the Global Architect: The most striking finding is that the global organization of these embeddings is driven by the Part-of-Speech (POS) of the context word, not the word being represented. For example, all relations that typically connect to a noun cluster together, regardless of whether the target word is a verb or an adjective.
  • Syntactic Senses: Dependency relations act as natural filters for polysemy. A word like "bank" shifts its neighborhood from financial terms (insurance, mortgage) when used in a compound relation to geographical terms (river, valley) when used in an obl (oblique) relation.
  • PPMI is More Consistent: Count-based models (PPMI) showed significantly higher geometric consistency across different relations than predictive models (SGNS). This suggests that SGNS might be more sensitive to the specific distributional quirks of individual relations.
  • Context Vectors are Underrated: In SGNS models, the "context" vectors (often discarded after training) actually retained more benchmark-relevant semantic information for most relations than the specifically trained "target" vectors in the reversed configuration.
Figure 1: t-SNE projection.

Key Results

  • Global Alignment: There was a strong correlation ($\rho$ up to .603) between the distance of relation centroids and the differences in their context-POS distributions. In contrast, correlations with the target word's own POS were weakly negative.
  • Local Neighborhoods: While broader lexical structures are preserved across relations, immediate "nearest neighbors" vary wildly. The chance-corrected Jaccard overlap for the top 10 neighbors across relations was low (under 0.15 for all models), highlighting how much the local semantic neighborhood changes with syntax.
  • Cross-POS Differentiation: The models successfully separated different POS usages of the same word. For instance, "fine" as an adjective (amod) clustered with "superb" and "exquisite," while "fine" as a noun (obj) clustered with "tax" and "fee."
Table 5: Cross-POS examples. 5 neighbours are shown in decreasing cosine-similarity order.

Limitations and Future Work

The authors identify two main limitations:

  1. Descriptive Nature: The analysis of local geometric consistency is largely descriptive; the authors suggest that future work could benefit from more direct statistical significance testing of these structures.
  2. Manual Polysemy Analysis: The qualitative analysis of word senses relied on manually selected examples because current lexical similarity benchmarks aren't designed to evaluate sense distinctions tied specifically to syntactic relations.

Future research could explore how these structured semantic spaces can be leveraged for downstream tasks like Word Sense Disambiguation or improving the interpretability of large language models.

Summary updated

Abstract

Dependency-based distributional models capture syntactically informed lexical similarities, but the geometric organisation of dependency-conditioned lexical representations remains underexplored. We investigate the global and local geometry of lemma-dependency relation embeddings using four representations trained on WikiText-103: two count-based PPMI configurations and two Skip-Gram with Negative Sampling (SGNS) configurations. At the global level, we analyse dependency-relation centroids, control for lexical identity through within-lemma centring, and examine their alignment with part-of-speech (POS) distributions. At the local level, we evaluate cross-relation geometric consistency, cross-method agreement, lexical similarity, and polysemous usages. Results show that global geometry is structured strongly by context POS across all representations. Locally, broader lexical similarity structures are partially preserved across dependency relations, while immediate neighbourhoods vary substantially, highlighting different usages. PPMI-based representations exhibit greater cross-relation consistency than SGNS-based representations, with cross-method agreement varying by relation. These findings reveal systematic linguistic organisation at multiple geometric scales and its sensitivity to representation choices.
BibTeX
@misc{phantat2026linguistic,
  author = {Bach Phan-Tat and Tung Kieu and Kris Heylen and Dirk Geeraerts and Stefano De Pascale and Dirk Speelman},
  title = {{The Linguistic Structure of Dependency-Conditioned Embeddings}},
  year = {2026},
  eprint = {2611.25909},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25909},
}
Figure