When training classic word embeddings like Word2Vec or GloVe, the standard approach relies on a linear sliding window. To learn the meaning of a target word, the model looks at a fixed number of words immediately preceding and following it.
While computationally efficient, this spatial proximity strategy operates on a flawed linguistic assumption: that words close to each other in a sentence are inherently related, and words far apart are not. In reality, human language is structured hierarchically, not linearly.
To bridge this gap, researchers developed dependency-based word embeddings. Instead of defining a word's context by a flat spatial window, this approach defines context using the grammatical relationships derived from a syntactic dependency parse tree.
Linear Window (Size 2):
[The quick] BROWN [fox jumps] over the lazy dog.
Dependency Context:
amod nsubj
[quick] ←── BROWN ──→ [fox]
As explored by DongLai ZHONG et al. in 基于非线性全局上下文的词嵌入, selecting context words based on functional syntactic relationships rather than simple spatial proximity improves the accuracy of semantic representations. A small set of grammatically relevant neighbors provides a much cleaner, more informative signal for learning word meanings than a larger window of adjacent but structurally unrelated words.
Topical vs. Functional Similarity
What happens to a vector space when you swap linear windows for syntactic dependencies? The core intuition is that it fundamentally changes the type of similarity the model captures.
In a seminal paper, Omer Levy et al. (Dependency-Based Word Embeddings) demonstrated that this shift moves embeddings from capturing topical similarity to capturing functional similarity:
- Topical Similarity (Linear Windows): Words are grouped together if they belong to the same domain or theme. For example, cup might cluster with coffee, tea, or kitchen. They frequently appear near each other, but they cannot substitute for one another.
- Functional Similarity (Dependency Windows): Words are grouped together if they share the same grammatical and semantic roles. Under this paradigm, cup clusters with mug, glass, or bowl. They share the same modifiers (empty cup, empty mug) and receive the same actions (drink from a cup, drink from a mug).
This distinction makes dependency-based models exceptionally good at identifying co-hyponyms (words that share a common superordinate category, like doctor and surgeon) rather than meronyms or topical associates (like doctor and hospital).
This property is highly valuable for tasks like lexical substitution. As shown by Omer Levy et al. in A Simple Word Embedding Model for Lexical Substitution, defining context through syntactic dependencies allows embeddings to better distinguish between words that merely appear in similar topics and words that can actually substitute for one another in a sentence. This is further supported by Felix Hill et al. in their evaluation of SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation, which concluded that grammatical dependency-based contexts are significantly more effective at capturing genuine semantic similarity than simple sliding-window approaches.
The Mechanics of Dependency Contexts
To build a dependency-based embedding model, a sentence is first processed by a dependency parser to identify its structural head-modifier relations.
In a standard skip-gram model, given a target word $w$ and a window size $k$, the training pairs are simply $(w, w_{-k}), \dots, (w, w_{+k})$. In a dependency-based skip-gram model, for a target word $w$ with modifiers $m_1, \dots, m_k$ and a head $h$, the contexts are pairs of the connected word and the explicit grammatical relation ($r$).
For example, in the phrase "scientists discover truth", the word discover has a subject relation (nsubj) with scientists and a direct object relation (dobj) with truth. The context pairs generated for discover would be:
$$\text{(scientists, nsubj)}, \quad \text{(truth, dobj)}$$
Conversely, the context for the noun scientists includes its head and the inverse relation: (discover, nsubj⁻¹).
By treating different grammatical relationships as distinct sources of information, the model learns how specific roles constrain meaning. Tong Wang 0012 et al. (Learning Lexical Embeddings with Syntactic and Lexicographic Knowledge) showed that factorizing specific syntactic dependency types (such as adjective-noun modifiers) allows models to learn high-quality functional semantic similarities even when trained on much smaller datasets.
Disentangling Roles: Governors and Dependents
Because words act as both operators (governors) and arguments (dependents) within a sentence, collapsing all occurrences into a single static vector can muddy the functional representation. To preserve this nuance, some architectures explicitly split word features based on their structural positioning.
Jinfeng Yang et al. (Representing Words as Lymphocytes) proposed a dual-context approach that splits word features into two distinct vectors: one capturing properties when the word acts as a governor and another for when it acts as a dependent. This prevents the loss of functional nuance that occurs when collapsing different syntactic roles into a single embedding. Similarly, Stefan Thater et al. (Ranking Paraphrases in Context) utilized a dual-layer vector space model that distinguishes between the semantic profile of a word as an argument and its profile as a predicate, allowing for precise context-sensitive adjustments when words interact.
Why and Where It Works
By filtering out the noise of accidental proximity and focusing on structural dependencies, these embeddings provide unique advantages across several downstream natural language processing tasks.
1. Handling Syntactic Invariance
Traditional sliding-window models struggle when structural transformations alter word order without changing the underlying meaning (e.g., active vs. passive voice: "The chef cooked the meal" vs. "The meal was cooked by the chef"). Because dependency graphs map the underlying grammatical relations identically in both cases, dependency-based representations are inherently robust to these shifts. For instance, Christian Hadiwinoto et al. (A Dependency-Based Neural Reordering Model for Statistical Machine Translation) demonstrated that pre-training embeddings to capture syntactic roles ensures that a model starts with a foundational understanding of structural relationships, which is essential for handling word reordering across languages.
2. Bridging Grammatically Divergent Languages
When mapping multiple languages into a shared bilingual vector space, linear proximity is a poor proxy for meaning if the language pairs have vastly different word orders (such as an SVO language like English and an SOV language like Japanese). Linli Xu et al. (Enhancing Semantic Representations of Bilingual Word Embeddings with Syntactic Dependencies) discovered that the performance advantage of dependency-based embeddings over window-based models increases in direct proportion to the grammatical divergence between language pairs, proving that structural context is critical for cross-lingual alignment.
3. Detecting Semantic Discordance
Tasks like sarcasm detection rely heavily on identifying subtle clashes between structurally related words (e.g., "I love getting flat tires"). Aditya Joshi 0001 et al. (Are Word Embedding-based Features Useful for Sarcasm Detection?) leveraged dependency-based contexts to capture how structurally related words interact semantically, allowing classification models to better isolate the "semantic discordance" that signals irony.
Limitations and Cross-Lingual Nuances
Despite their strengths, dependency-based embeddings are not a universal replacement for linear models. Because they rely on an explicit preprocessing step, their quality is strictly bounded by the accuracy of the upstream dependency parser.
Furthermore, the utility of syntactic context is not uniform across all languages. Ivan Vulić et al. (Is "Universal Syntax" Universally Useful for Learning Distributed Word Representations?) revealed a notable Anglo-centric bias in these models. While language-agnostic universal parses are highly effective for English, their advantage over simple bag-of-words models diminishes in morphologically rich languages like German and Italian. This suggests that the benefits of dependency-based context selection depend heavily on the specific typological characteristics of the language being modeled.
Key Foundational Work
Automatic Thesaurus Construction based on Grammatical Relations (1995) — contributed 1 concept elements to this direction.