Standard natural language models often view text as a linear sequence of tokens. While this "bag-of-words" or "sliding window" approach captures local patterns, it is fundamentally nearsighted. In human language, the most important relationship for a word—such as a verb and its subject—might be separated by a dozen intervening adjectives, clauses, or parentheticals.
To overcome this, researchers have developed methods to extract structural features from syntactic parse trees. By shifting from linear proximity to hierarchical distance, models can "short-circuit" the sentence, linking words based on their functional roles rather than their physical position.
The Structural Neighborhood
In a linear model, a word's context is its immediate neighbors. In a structural model, the context is defined by the word's position in a dependency or constituent tree. This "structural neighborhood" typically includes the word’s parent (its syntactic head), its children (its modifiers), and its siblings.
As Adam L. Berger and Harry Printz noted in A Comparison of Criteria for Maximum Entropy/ Minimum Divergence Feature Selection, these structural features offer higher precision because they only activate when a specific linguistic connection is present. For example, to determine if a word is a "Subject," a model might look at the part-of-speech tag of its parent node. Yi Zheng and colleagues applied this in Hedge Classification with Syntactic Dependency Features Based on an Ensemble Classifier, using the hierarchical role of a token to distinguish between a literal statement and a "hedged" expression of uncertainty.
This neighborhood can be expanded to capture even broader context. Eliyahu Kiperwasser and Yoav Goldberg demonstrated in Semi-supervised Dependency Parsing using Bilexical Contextual Features from Auto-Parsed Data that including the immediate neighbors of both the head and the modifier (triplets) helps disambiguate relationships that are unclear from the words alone.
Bridging Distance with Syntactic Paths
When two related words are far apart in a sentence, the linear distance makes it difficult for a model to correlate them. However, in a parse tree, the syntactic path—the sequence of nodes one must traverse to get from one word to another—is often very short.
In A Joint Model for Semantic Role Labeling, Aria Haghighi, Christopher D Manning, and Kristina Toutanova used specialized path features to bridge the gap between predicates and their arguments. These paths project through complex syntactic structures, like auxiliary verb chains, to identify "who did what to whom" even when the actors are separated by multiple clauses.
This mechanism is particularly effective for:
- Error Correction: Wei Xu and Joel R. Tetreault showed in Exploiting Syntactic and Distributional Information for Spelling Correction with Web-Scale N-gram Models that dependency paths allow models to capture informative context that is "nearsighted" n-gram windows would miss.
- Disambiguation: For acronyms or ambiguous terms, following the structural branches of a sentence allows a model to link a term directly to its descriptor, regardless of physical distance, as explored by Amir Pouran Ben Veyseh in What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation.
Structural Features in Neural Architectures
While early work focused on discrete features for statistical models, the intuition has successfully transitioned to neural networks and embeddings. By incorporating structural information, models can learn more nuanced representations of word meaning.
In Good, Better, Best: Choosing Word Embedding Context, Bing Xiang and Bowen Zhou demonstrated that embeddings trained on dependency-based contexts are significantly better at capturing functional and taxonomic similarities than those trained on linear windows. Similarly, James Henderson and Majid Yazdani used dependency-based features to build context-aware vectors for A Model of Zero-Shot Learning of Spoken Language Understanding, allowing the model to generalize to semantic labels it had never seen during training.
Why It Works: The Hierarchy of Meaning
The reason these features work across such diverse tasks—from Chinese POS Tagging by Wanxiang Che to Prosodic Phrase Prediction by Guoping Hu—is that they align the model's "vision" with the actual structure of human thought.
Language is not a flat string; it is a nested hierarchy. By extracting features like ceiling nodes for pronoun resolution (Hwee Tou Ng and Shanheng Zhao) or parent-sibling identities for machine translation (Toshiaki Nakazawa and Sadao Kurohashi), researchers provide models with a structural map. This map allows the system to ignore the "noise" of linear distance and focus on the grammatical and semantic skeleton that defines a sentence's meaning.