Minimalist Feature Engineering for Text Classification

124 concepts102 papers1 proposal

In contemporary natural language processing, the dominant paradigm favors scale: larger models, denser architectures, and high-dimensional embeddings. The underlying assumption is that complex linguistic phenomena require equally complex representations to be accurately captured.

However, a substantial body of research demonstrates a compelling counter-intuition: minimalist feature engineering—the practice of using small, highly optimized, and often shallow feature sets—can match or even outperform complex, high-dimensional models. By drastically reducing the feature space, researchers can bypass the curse of dimensionality, eliminate noise, and build models that generalize better to unseen data, all while maintaining computational efficiency and interpretability.

The Paradox of High-Dimensional Noise

Why does reducing the number of features often improve a model's performance? The answer lies in the relationship between dimensionality, noise, and generalization. In high-dimensional spaces, data points become sparse, making it easy for classifiers to find spurious correlations that do not reflect true linguistic patterns.

When the feature space is aggressively pruned to include only the most discriminative indicators, the model is forced to rely on robust, invariant signals. For example, in machine translation evaluation, Shuqi Sun et al. demonstrated in A Re-examination on Features in Regression Based Approach to Automatic MT Evaluation that a regression model utilizing just six carefully selected linguistic features could match the human-judgment correlation of complex 53-feature models. Crucially, when tested on entirely new datasets, the minimalist model actually outperformed its high-dimensional counterpart, proving that parsimony directly aids out-of-domain generalization.

This phenomenon is not unique to translation evaluation. In the domain of hedge detection (identifying uncertainty in scientific text), Erik Velldal showed in Detecting uncertainty in biomedical literature: a simple disambiguation approach using sparse random indexing that combining a linear Support Vector Machine (SVM) with Sparse Random Indexing compressed the feature space by two orders of magnitude. Despite this massive reduction, the system achieved state-of-the-art results, demonstrating that the discriminative power of local context is often far more critical than global structural analysis. Similarly, in unsupervised event coreference resolution, Cosmin Adrian Bejan discovered in Unsupervised Event Coreference Resolution with Rich Linguistic Features that generative models could maintain competitive performance while utilizing as little as 1.5% of available feature values, effectively ignoring the surrounding statistical noise.

Shallow Features vs. Dense Embeddings

With the advent of deep learning, dense representations like BERT embeddings became the default choice for text classification. Yet, dense embeddings carry thousands of dimensions that compress broad semantic information, much of which may be irrelevant to a specific, targeted task.

For specialized classification challenges, simple surface-level features (such as character n-grams or TF-IDF representations) combined with linear classifiers frequently prove superior.

[Raw Text] ───> [Exhaustive BERT Embedding] ───> 768+ Dimensions (High Noise/Overfitting)
[Raw Text] ───> [Targeted Surface N-grams]  ───> Few Optimized Dimensions (High Signal)

Consider the task of predicting the difficulty of medical exam questions. Rishikesh Fulari et al. explored this in Utilizing Machine Learning to Predict Question Difficulty and Response Time for Enhanced Test Construction, finding that simpler models leveraging focused, handcrafted linguistic features outperformed complex, high-dimensional BERT embeddings. The dense embeddings suffered from the curse of dimensionality, whereas the targeted features isolated the exact syntactic and lexical markers that make a question difficult for humans.

A similar pattern emerges in social media moderation and dialect identification, where text is often short, informal, and highly variable:

By avoiding dense neural architectures, these systems bypass extensive preprocessing and heavy computational overhead while capturing the exact lexical "fingerprints" necessary for classification.

The Sufficiency of Local and Morphological Signals

The success of minimalist feature engineering stems from a fundamental truth about language: the core signal for many tasks is highly concentrated. Deep linguistic parsing—such as building complete dependency trees or performing exhaustive part-of-speech tagging—often yields diminishing returns once a baseline level of surface feature extraction is achieved.

A striking example of this sufficiency is found in morphologically rich languages like Turkish. In Categorization of Turkish News Documents with Morphological Analysis, Burak Kerim Akkus investigated whether computationally expensive morphological analysis improved text categorization. The author discovered that a simple heuristic—truncating every word to its first five characters—performed just as well as, and sometimes better than, deep linguistic parsing. The first five characters captured the semantic root of the word while discarding complex suffix variations, providing a "cheap" yet highly effective feature space compression.

This principle applies across a variety of structural NLP tasks:

  • Feature Selection: In OffTamil@DravideanLangTech-EASL2021: Offensive Language Identification in Tamil Text, Disne Sivalingam et al. utilized $\chi^2$ feature selection to isolate a highly restricted set of discriminative word unigrams, achieving competitive accuracy while streamlining the model.
  • Coreference Resolution: Early breakthroughs in machine learning for coreference resolution relied on this exact parsimony. Wee Meng Soon et al. demonstrated in Corpus-Based Learning for Noun Phrase Coreference Resolution that a small subset of shallow features—such as basic string matching, gender agreement, and relative physical distance—accounted for the bulk of predictive performance, rendering complex semantic features largely redundant.

High-Density Structural Proxies

When structural or semantic relationships are required, minimalist engineering focuses on injecting high-density proxies rather than allowing a model to navigate raw, unrefined data. Instead of forcing a network to learn complex hierarchies from scratch, researchers can expose a handful of highly informative, engineered signals.

For instance, recognizing implicit discourse relations (how two sentences relate when transition words like "but" or "because" are missing) is notoriously difficult. In The Effects of Discourse Connectives Prediction on Implicit Discourse Relation Recognition, Yu Xu solved this by using a language model to predict the single most likely missing connective, and then using that single predicted word as the primary feature for classification. This minimalist approach matched the performance of models utilizing thousands of complex statistical features, because the semantic link between the sentences was successfully concentrated into a single proxy word.

Even in parsing itself, where structure is the ultimate goal, minimalism holds. James Cross 0003 et al. showed in Incremental Parsing with Minimal Features Using Bi-Directional LSTM that if the underlying context representations are sufficiently rich, a state-of-the-art greedy parser can be built using as few as three positional features.

Ultimately, whether ranking community questions using basic word importance levels, as shown by Surya Agustian in UINSUSKA-TiTech at SemEval-2017 Task 3: Exploiting Word Importance Levels for Similarity Features for CQA, or extracting temporal relations using basic morpho-syntactic tags, as shown by Steven Bethard in ClearTK-TimeML: A minimalist approach to TempEval 2013, the intuition remains identical. By identifying and isolating the core linguistic drivers of a task, minimalist feature engineering creates models that are faster to train, easier to interpret, and remarkably resilient to overfitting.

Key Foundational Work

Recognizing Text Genres With Simple Metrics Using Discriminant Analysis (1994) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_minimalist_feature_engineering_39429727,
  title = {minimalist feature engineering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimalist-feature-engineering/rdr_3942972782e4427181b27115fb6da6d5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_minimalist_feature_engineering_39429727_v1,
  title = {minimalist feature engineering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimalist-feature-engineering/rdr_3942972782e4427181b27115fb6da6d5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake