End-to-End Feature Selection using Continuous Relaxations

This direction is split into other directions: differentiable feature selection. Its last version stays readable and citable here.

Version 1 of 184 papers92 conceptsfrozen 2026-10-04

Feature selection is traditionally a "hard" problem—literally. In a dataset with $d$ features, finding the optimal subset of size $k$ requires searching through $\binom{d}{k}$ combinations. This combinatorial explosion makes exhaustive search impossible for high-dimensional data like genomics or sensor streams. Historically, researchers relied on "filter" methods (ranking features by simple statistics) or "wrapper" methods (greedily adding features one by one).

However, a growing body of research has moved toward Differentiable Feature Selection. By relaxing the discrete "yes/no" choice of a feature into a continuous "how much" weight, we can use backpropagation to learn the optimal feature subset simultaneously with the model's parameters.

From Discrete Toggles to Differentiable Dimmers

The core challenge of feature selection is that the operation of "picking" a feature is non-differentiable. If you multiply a feature by 1 to keep it or 0 to discard it, the gradient is zero almost everywhere. You cannot "nudge" a 0 toward a 1 using standard gradient descent.

To solve this, researchers like Jason Weston and Sayan Mukherjee early on proposed reformulating feature selection as a continuous optimization task. In Feature Selection for SVMs, they introduced real-valued scaling vectors that weight individual input dimensions, allowing gradient-based methods to identify relevant features in high-dimensional DNA microarray data.

The modern evolution of this idea involves stochastic relaxations. Instead of a fixed weight, we treat the selection of a feature as a random variable. To make this trainable, we use the "reparameterization trick," most notably via the Concrete distribution (or Gumbel-Softmax).

The Concrete Selector Layer

In Concrete Autoencoders, Muhammed Fatih Balin and James Y. Zou introduced a "selector layer" that uses a continuous relaxation of categorical distributions.

The intuition is simple:

  1. Start Soft: Early in training, the model looks at a "weighted average" of all features. The selection mask is "blurry," with many features contributing a small amount.
  2. Anneal: A "temperature" parameter is gradually lowered. As the temperature drops, the distribution pushes the weights toward a "one-hot" vector—effectively forcing the model to pick a single feature.
  3. End Hard: By the end of training, the "dimmer switch" has become a "toggle switch," and the model has converged on a discrete subset of the original features.

This approach allows for global feature selection that scales to tens of thousands of dimensions. It has been successfully applied to hardware optimization, such as in End-to-end learnable EEG channel selection, where Thomas Strypsteen and Alexander Bertrand used it to let a network decide which physical sensors are most important for a task.

Instance-Wise Selection and Explainability

While global selection finds features that are important for the entire dataset, Instance-Wise Feature Selection (IWFS) identifies which features matter for a specific prediction.

In Learning to Explain (L2X), Jianbo Chen and Le Song framed this as an information-theoretic problem. They used a "selector network" to output the parameters of a Concrete distribution, which then masks the input for a "predictor network." By maximizing the mutual information between the selected features and the label, the model learns to find the most informative "smoking gun" features for every individual case.

This same logic is used in AI2TALE to localize phishing attacks by identifying deceptive sentences, and in PVAR to refine adversarial attacks by identifying the most "vulnerable" pixels in an image.

Beyond Simple Masks: Structural and Rank-Based Selection

Recent work has expanded the differentiable toolkit beyond simple binary masks.

  • Rank-Based Objectives: Krunoslav Lehman Pavasovic and colleagues developed difFOCI, a differentiable version of rank-based statistical measures. This allows neural networks to select features based on their statistical dependency while ignoring spurious correlations.
  • Structural Constraints: In GRAND-SLAMIN’, Shibal Ibrahim uses a "smooth-step" relaxation to enforce hierarchy and sparsity in Generalized Additive Models, ensuring that the model only considers complex feature interactions if the individual features themselves are already deemed important.
  • Causal Invariance: Methods like RAMEN use Gumbel-Softmax relaxations to search for "invariant" feature sets—variables whose relationship with the outcome remains stable across different environments, which is a hallmark of causal relationships.

Why It Works: The Power of Joint Optimization

The primary reason these methods outperform traditional techniques is joint optimization. In a traditional pipeline, you select features first and then train your model. If your feature selection step misses a non-linear interaction that the model could have used, that information is lost forever.

By making selection differentiable, the model can say: "I'm struggling to minimize my loss; let me try shifting my attention to Feature B instead of Feature A." As shown in PROPOSE by Ian Covert and Su-In Lee, this leads to the discovery of non-redundant, highly informative feature panels that achieve high accuracy even with extremely small subset sizes.

Whether it's choosing genes for spatial transcriptomics or reducing congestion in marketplaces by managing item representations, the shift from discrete search to continuous relaxation has turned feature selection from a bottleneck into a learnable component of the neural architecture.

Papers in this version

Showing 60 of 84 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_end_to_end_feature_selection_using_conti_0074940f,
  title = {End-to-End Feature Selection using Continuous Relaxations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-feature-selection-using-continuous-relaxations/rdr_0074940f8f82403797166f64a10e0d07},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_end_to_end_feature_selection_using_conti_0074940f_v1,
  title = {End-to-End Feature Selection using Continuous Relaxations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/end-to-end-feature-selection-using-continuous-relaxations/rdr_0074940f8f82403797166f64a10e0d07/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}