Instance-Wise Feature Selection

83 concepts65 papers1 proposal

Traditional feature selection methods operate under a "one-size-fits-all" assumption: they identify a single, fixed subset of variables that are deemed universally important across an entire dataset. While this global approach simplifies models and reduces overfitting, it fails to account for the inherent heterogeneity of real-world data. In complex domains like personalized medicine, financial forecasting, or multi-modal diagnostics, the specific features driving an outcome can vary drastically from one individual sample to the next.

For instance, a clinical biomarker that is highly predictive for one patient demographic might be entirely irrelevant or noisy for another. As noted by Jinsung Yoon et al. in their work on INVASE, global selection averages out these critical subpopulation signals, forcing an unnecessary trade-off where a model is either too complex for simple cases or too simple for difficult ones.

To resolve this, a powerful paradigm has emerged: instance-wise feature selection (also known as sample-specific or personalized feature selection). Instead of finding a single global subset, instance-wise selection dynamically identifies a unique subset of relevant variables for each individual data point.


The Core Mechanism: Amortized Gating and Masking

How can a model efficiently determine which features matter for an unseen sample? The most common approach relies on a dual-network architecture: a selector (or gating) network and a predictor network.

Input Vector (x) ───► [ Selector Network ] ───► Instance-Specific Mask (m)
       │                                                    │
       └───────────────────► ( Element-wise Mult. ) ◄───────┘
                                    │
                                    ▼
                         Filtered Input (x ⊙ m)
                                    │
                                    ▼
                           [ Predictor Network ] ───► Prediction (y)

Instead of optimizing a static mask during training, the selector network learns a function that maps an individual input vector to a customized feature mask. This mask is applied via element-wise multiplication to filter out irrelevant features before they reach the primary classifier or regressor.

Different frameworks implement this gating mechanism in distinct ways:

  • Stochastic Gates: In Locally Sparse Neural Networks for Tabular Biomedical Data, Junchen Yang utilizes sample-specific stochastic gates combined with a stability-inducing kernel. This allows the network to zero out irrelevant inputs dynamically, preventing overfitting on small, high-dimensional biomedical datasets.
  • Reinforcement Learning Agents: Alvaro H. C. Correia proposes a framework where a reinforcement learning agent acts as the selector, generating a binary mask for each individual input to minimize model loss while incorporating human intuition into the learning loop.
  • Copula-Based Dependency Modeling: To account for real-world feature correlations (such as height and weight), Hanyu Peng et al. introduce a framework using Gaussian copulas to explicitly model feature dependencies, generating a more precise instance-wise selection mask that avoids selecting redundant variables.

Sequential Selection and Active Acquisition

When features are not just irrelevant but also costly to acquire (e.g., expensive medical tests), instance-wise selection transforms into a sequential decision-making problem. The model must decide step-by-step which feature to inspect next, based on the values of the features it has already observed.

In Datum-Wise Classification: A Sequential Approach to Sparsity, Gabriel Dulac-Arnold et al. frame this as a reinforcement learning task. The agent sequentially selects features for a specific data point and chooses to "stop looking" the moment it has accumulated enough information to make a confident prediction.

This sequential logic is highly effective for balancing accuracy against data collection budgets. Jaromír Janisch et al. formalize this as Classification with Costly Features (CwCF), using Deep Reinforcement Learning to dynamically prioritize the next feature to acquire based entirely on the history of prior observations. Similarly, Chao Ma et al. introduce EDDI, which pairs a Partial VAE (capable of handling arbitrary missing data) with an information-theoretic acquisition function to personalize feature discovery on the fly.


Information-Theoretic and Contextual Selection

Rather than relying purely on reinforcement learning heuristics, several prominent methods frame instance-wise selection as an information-theoretic optimization problem. The goal is to find a subset of features for a given instance that maximizes the explanation of the model's response.

In the L2X (Learning to Explain) framework, Jianbo Chen et al. optimize a global explainer network to maximize the mutual information between the selected local features and the model's prediction. To bypass the training instabilities and computational overhead often associated with reinforcement learning and generative explainer networks, Ian Connick Covert et al. propose an amortized optimization approach that trains a neural network to greedily maximize Conditional Mutual Information (CMI), providing fast, stable, and dynamic feature selection.

Bridging the Gap in Tabular Data

In tabular data modeling, deep neural networks have historically lagged behind tree-based models (like XGBoost) because trees naturally discover sample-specific decision paths. Modern architectures use instance-wise feature selection to close this gap.

For example, Kuan-Yu Chen et al. developed Trompt, an architecture that uses prompt embeddings to query static column embeddings. This allows the model to dynamically adjust its focus on specific features based on the unique characteristics of each input row. In a similar vein, Xinhao Zhang 0001 et al. introduce TFWT (Tabular Feature Weighting with Transformer), leveraging self-attention mechanisms to assign unique importance weights to features based entirely on sample-specific context.


Why It Works: The Intuition

Instance-wise feature selection succeeds because it shifts the feature selection bottleneck from the dataset level to the sample level.

  1. Capturing Subpopulation Heterogeneity: By allowing the feature mask to change per instance, the model can preserve localized signals. As shown by Aria Masoomi et al., dynamically grouping and selecting features per instance prevents subtle, idiosyncratic biomarkers from being averaged out by global filters.
  2. Granular Interpretability: Instead of providing a generic overview of what a model values across thousands of samples, instance-wise selection provides a precise feature relevance profile for every single instance. As Jan Philip Göpfert et al. demonstrate with locally adaptive metrics, this reveals exactly how feature importance shifts as a data point moves across different regions of the data distribution.
  3. Efficiency and Cost Reduction: By focusing data collection and computational capacity only on the variables that drive variance for a specific profile, these systems achieve target accuracy levels using a fraction of the total available features, drastically reducing the cost and burden of data collection.

Key Foundational Work

Sequential Feature Selection for Classification (2011) — contributed 2 concept elements to this direction.
Datum-Wise Classification: A Sequential Approach to Sparsity (2011) — contributed 1 concept elements to this direction.
Dependency Bagging (2005) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 65 papers
  2. Version 1 · 2026-10-04 · 87 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_instance_wise_feature_selection_cb8690ee,
  title = {instance-wise feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instance-wise-feature-selection/rdr_cb8690ee9c33410abeece1f783a6de37},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_instance_wise_feature_selection_cb8690ee_v2,
  title = {instance-wise feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/instance-wise-feature-selection/rdr_cb8690ee9c33410abeece1f783a6de37/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake