When designing natural language processing systems, it is intuitive to assume that layering explicit linguistic knowledge—such as part-of-speech (POS) tags, dependency trees, or semantic roles—will inherently improve a model's performance. After all, human language relies on these structural rules to convey meaning.
However, a recurring pattern across decades of NLP research demonstrates a surprising counter-intuitive reality: explicit syntactic and semantic features are frequently redundant. When models have access to raw lexical data, sequential context, or dense representations, they implicitly capture these structural relationships, rendering highly engineered linguistic annotations largely unnecessary.
Lexical Subsumption: How Words Encode Their Own Syntax
Why do explicit grammatical labels often fail to provide incremental value? The primary reason is that the identity and local context of words themselves already carry their structural behavior.
In an early information-theoretic analysis, Wu et al. demonstrated in An Information-Theoretic Empirical Analysis of Dependency-Based Feature Types for Word Prediction Models that specific lexical identities are vastly more informative for word prediction than abstract grammatical labels like POS tags or dependency types. The words themselves implicitly signal their grammatical relationships. This finding is mirrored in modern parsing challenges; Müller et al. showed in Dependency parsing with latent refinements of part-of-speech tags that parsers frequently struggle with highly granular linguistic tags because those distinctions merely replicate patterns already captured by lexicalized models.
This lexical sufficiency is evident across diverse tasks:
- Speech and Text Segmentation: In conversational speech processing, Gupta et al. found in Extracting Clauses for Spoken Language Understanding in Conversational Systems that word-level contextual features capture enough structural information to make explicit POS tags completely redundant.
- Domain-Specific Text: When analyzing non-standard or ungrammatical text, such as clinical narratives, Roberts et al. showed in Extracting Clinical Relationships from Patient Narratives that detailed POS tagging can introduce harmful noise, whereas surface-level lexical patterns are entirely sufficient.
- Text Classification: In Native Language Identification (NLI), Léger et al. observed in Feature Space Selection and Combination for Native Language Identification that simple lexical n-grams capture enough stylistic nuance that complex structural analysis adds little value. Dras et al. reinforced this in Contrastive Analysis and Native Language Identification and NLI Shared Task 2013: MQ Submission, finding that dense n-gram distributions implicitly subsume explicit syntactic error markers and complex syntactic features.
- Identity Resolution: Even for high-level semantic tasks like web people search, Artiles et al. demonstrated in The role of named entities in Web People Search that explicit named-entity features are largely redundant when simple word n-grams are available.
The Overlap Between Syntactic and Semantic Layers
Another major source of redundancy occurs at the intersection of syntax and semantics. Because semantic roles (e.g., agent or patient) heavily correlate with specific syntactic constituents (e.g., subject or object), explicitly modeling both layers often results in overlapping signals.
For example, Li et al. observed in A Unified Model for Soft Linguistic Reordering Constraints in Statistical Machine Translation that integrating semantic role information into translation models provides diminishing returns when robust syntactic constraints are already active. The syntactic model effectively captures the signal intended for the semantic model. Conversely, local contextual features like POS tags can provide enough structural scaffolding for semantic tasks without full syntactic trees; Martins et al. showed in Priberam: A Turbo Semantic Parser with Second Order Features that high-quality semantic graphs can be generated with minimal reliance on external syntactic parsers.
This mutual information means that higher-level semantic features can also absorb lower-level syntactic needs, and vice versa:
- Yao et al. demonstrated in Recurrent neural networks for language understanding that standard POS tags offer minimal performance gains in semantic labeling once named-entity information is present.
- Kong et al. found in Employing the Centering Theory in Pronoun Resolution from the Semantic Perspective that accurate semantic role features largely subsume the structural information provided by complex pronominal ranking hierarchies.
- Che et al. noted in Jointly Modeling WSD and SRL with Markov Logic that explicit noun argument senses yield diminishing returns because lower-level features like lemmas and POS tags provide sufficient information.
Sequential Models and the Obsolescence of Explicit Trees
With the advent of powerful sequential and graph-based architectures, the need for engineered structural features has decreased even further. Neural networks excel at extracting hierarchical context directly from sequential or positional representations.
Consider dependency parsing itself. de Lhoneux et al. investigated in Recursive Subtree Composition in LSTM-Based Dependency Parsing whether explicitly modeling tree structures via recursive composition adds value to BiLSTM-based parsers. The authors discovered that recursive modeling is largely redundant because the sequential BiLSTM already implicitly captures the necessary hierarchical context.
Explicit Tree Features (Recursive Composition, Sibling Relations)
│
▼ (Overlapping Context / Redundant Signal)
┌──────────────────────────────────────────────────────────┐
│ Sequential Context (BiLSTMs, Relative Positions, Spans) │
└──────────────────────────────────────────────────────────┘
▲
│ (Implicitly Subsumes)
Word Identities & Local Lexical Context
Furthermore, complex graph features often duplicate simpler positional or span-based information:
- Position Embeddings: Mensah et al. showed in An Empirical Study on Leveraging Position Embeddings for Target-oriented Opinion Words Extraction that graph-based syntactic dependency features offer only marginal improvements over simple relative position embeddings, suggesting that local sequential context dominates the task.
- Span Representations: Yang et al. noted in Combining (Second-Order) Graph-Based and Headed-Span-Based Projective Dependency Parsing that incorporating headed span information renders traditional second-order features (like sibling relationships) redundant, as the span representations already encapsulate that structural context.
- Feature Overlap: In feature-engineered systems, Pradhan et al. observed in Semantic Role Labeling by Tagging Syntactic Chunks that complex engineered features like syntactic paths, clause positions, and relative distances capture highly overlapping structural information, making individual contributions difficult to isolate.
- Morphosyntactic Redundancy: Even within formal grammar frameworks, Sima'an et al. demonstrated in Modeling Morphosyntactic Agreement in Constituency-Based Parsing of Modern Hebrew that integrating head-tag information alongside explicit agreement features introduces redundant signals because agreement patterns frequently subsume structural head-tag information.
Why It Works: The Principle of Correlated Signals
The fundamental reason explicit syntactic features exhibit such high redundancy is that language is a system of highly correlated signals. A single structural relationship is mirrored across multiple levels simultaneously: in the specific choice of words (lexical), the order of those words (sequential), the grammatical categories (syntactic), and the underlying meaning (semantic).
When a model is trained on dense, high-dimensional representations—whether they are lexical n-grams or neural hidden states—it naturally exploits these correlations to optimize its objective. As a result, explicitly injecting a feature like a POS tag or a dependency path rarely provides a unique discriminative signal that the model has not already derived from simpler inputs. Recognizing this redundancy allows practitioners to build faster, leaner, and less complex models without sacrificing predictive accuracy.
Key Foundational Work
An Information-Theoretic Empirical Analysis of Dependency-Based Feature Types for Word Prediction Models (1999) — contributed 1 concept elements to this direction.
Extracting Clauses for Spoken Language Understanding in Conversational Systems (2002) — contributed 1 concept elements to this direction.