Diminishing Returns from Overlapping Feature Sets

27 concepts18 papers1 proposal

A common intuition when designing natural language processing models is that more information is inherently better. If feature set $A$ improves a model's accuracy, and feature set $B$ also provides a positive signal, it seems logical that combining $A$ and $B$ should yield an additive performance gain.

Yet, empirical research across a wide array of NLP tasks—ranging from automatic speech recognition to syntactic parsing—frequently encounters a frustrating plateau. When multiple high-performing feature sets are combined, they regularly yield diminishing returns. Understanding why this happens requires looking past the surface definitions of these features to see how they act as overlapping proxies for the exact same underlying linguistic phenomena.

The Root Cause: Latent Information Overlap

The core reason for non-additive performance gains is that features derived from entirely different linguistic levels or methodologies are often highly correlated. When two distinct heuristics capture the same structural, semantic, or phonological signal, a model cannot extract unique information from their combination.

Consider the task of computational word segmentation. John K. Pate et al. demonstrated this phenomenon in their paper Syllable weight encodes mostly the same information for English word segmentation as dictionary stress. While syllable weight (the structural "heaviness" of a syllable) and dictionary-defined stress are both powerful individual cues for identifying word boundaries in English, combining them fails to outperform a model using only the single most informative cue. Because the physical structure of a syllable already encodes its stress patterns, the two features are informationally redundant.

This pattern repeats across different tasks and domains:

Surface Features vs. Deep Models

This redundancy is particularly evident when researchers attempt to supplement rich, surface-level features with deep statistical or neural language models.

In A Cascaded Linear Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging, Wenbin Jiang et al. observed that character-based discriminative features often capture the exact same linguistic patterns as word-level language models. If a system already utilizes a rich set of character features, incorporating a word-level language model yields minimal performance improvements because the character patterns have already mapped the necessary context.

A similar effect occurs in biomedical entity classification. The authors of SimSem: Fast Approximate String Matching in Relation to Semantic Category Disambiguation found that the benefits of complex, approximate dictionary string matching largely disappear when standard span-internal features—such as character n-grams, prefixes, and suffixes—are already present in the classifier. The fuzzy dictionary lookups simply recapture the morphological variance already exposed by basic subword features.

Even modern, deep contextualized representations are subject to this law of diminishing returns. As shown by Mehrdad Yousefpoori-Naeim et al. in Using Machine Learning to Predict Item Difficulty and Response Time in Medical Tests, adding advanced language model embeddings like MPNet to a model that already includes rich text-cohesion and structural features offers only marginal improvements. The deep embeddings fail to provide a unique signal because the simpler linguistic metrics have already explained the variance associated with text difficulty.

Redundancy Across Semantic and Morphological Scales

When engineering features across different granularities—such as blending word-level semantics with subword morphology—the overlapping information space continues to limit marginal gains.

[Raw Text Input]
       │
       ├─► Feature Set A: Word-Level Embeddings ──► [Captures Semantic Context] ──┐
       │                                                                          ▼
       └─► Feature Set B: Topic Models (LDA) ────► [Captures Semantic Context] ──┴─► [Overlapping Information Space]
                                                                                               │
                                                                                               ▼
                                                                                   Diminishing Additive Gains

In automatic speech recognition (ASR) rescoring, Chang Liu et al. evaluated the combination of Latent Dirichlet Allocation (LDA) topics and word-embedding features in Semantic Features Based N-Best Rescoring Methods for Automatic Speech Recognition. While both semantic representations successfully reduce Word Error Rates (WER) individually, combining them does not lead to performance gains beyond the best individual feature set. Both methods map the same underlying semantic harmony of the sentence.

For morphologically rich languages, this redundancy appears when mixing word-level and morpheme-level n-grams. In Discriminative n-gram language modeling for Turkish, Ebru Arisoy et al. demonstrated that discriminative language models struggle to extract unique signals from overlapping word and morphological feature sets without a mechanism to dynamically select the most relevant context. This challenge is echoed by Thomas Müller et al. in A Comparative Investigation of Morphological Language Modeling for the Languages of the European Union, which notes that morphological features and distributional contexts provide heavily overlapping information during word clustering.

When More is Less: The Structural Cost of Redundancy

Feature overlap is not merely an efficiency issue; it can actively degrade model performance. When redundant features are introduced into a model without structural separation, they increase the dimensionality of the feature space, which can exacerbate data sparseness and lead to overfitting.

Khalil Sima'an et al. documented this hazard in Modern Hebrew constituency parsing. In Modeling Morphosyntactic Agreement in Constituency-Based Parsing of Modern Hebrew, the authors showed that combining head-tags with explicit morphosyntactic agreement markers degraded parsing accuracy. Because the agreement features already captured the necessary structural relations, adding redundant head-tags split the statistical counts, worsened data sparseness, and harmed the model's ability to generalize.

Similarly, in Native Language Identification, Yasmeen Bassas et al. found that combining lexical and unlexicalized features failed to consistently improve performance over the best individual feature sets in Investigating Linguistic Features for Arabic NLI, illustrating that naive feature concatenation often introduces noise rather than clarity.

Designing for Orthogonality

Recognizing the bottleneck of feature redundancy shifts the goal of system design from feature completeness to feature orthogonality. To achieve true synergistic gains, architectures must be engineered to prevent features from shadowing one another.

One approach is to use multi-stage or cascaded architectures that isolate different feature types. For example, Wenbin Jiang et al. mitigated redundancy by separating character-based decoding from word-level language model scoring into a two-layer cascaded linear model, allowing each layer to operate on manageable, distinct complexities.

Alternatively, discriminative reranking frameworks can be used to balance traditional base models with rich, non-overlapping global features. Michael White 0001 et al. utilized an averaged perceptron model in Perceptron Reranking for CCG Realization to successfully integrate traditional language model probabilities with discriminative syntactic features.

However, even in large-scale system ensembles, managing this balance remains a delicate task. As Markus Freitag et al. discovered in Combined spoken language translation, applying advanced language model rescoring to an ensemble yields diminishing returns if the constituent models have already utilized similar neural features internally.

Ultimately, when a component like the Verb Group Identifier built by Harshada Gune et al. in Verbs are where all the action lies: Experiences of Shallow Parsing of a Morphologically Rich Language underperforms expectations, it is often because existing morphological features have already quietly solved the problem. True progress in complex NLP tasks relies on identifying and engineering orthogonal signals, rather than stacking variations of the same underlying information.

Key Foundational Work

Discriminative n-gram language modeling for Turkish (2008) — contributed 3 concept elements to this direction.
A Cascaded Linear Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging (2008) — contributed 2 concept elements to this direction.
Perceptron Reranking for CCG Realization (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_feature_redundancy_diminishing_returns_f_5203536e,
  title = {feature redundancy (diminishing returns from feature combination)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-redundancy-diminishing-returns-from-feature-combination/rdr_5203536e92df477fba80b154797a8b08},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_feature_redundancy_diminishing_returns_f_5203536e_v1,
  title = {feature redundancy (diminishing returns from feature combination)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-redundancy-diminishing-returns-from-feature-combination/rdr_5203536e92df477fba80b154797a8b08/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake