Controlled Feature Selection in High Dimensions

43 concepts40 papers

Imagine trying to identify the genetic drivers of a rare disease by analyzing 50,000 genetic variants in a study of only 500 patients. If you run independent statistical tests or standard regressions across all these variants, pure statistical noise will inevitably yield hundreds of false positives.

Traditional regularized regression methods like the Lasso shrink coefficients to zero to enforce sparsity, but as noted by Krakovska et al., shrinkage-based linear selection frequently suffers from high false discovery rates (FDR), mistakenly identifying irrelevant variables alongside true predictors. To solve this, modern high-dimensional statistics relies on Controlled Feature Selection—a suite of frameworks designed to ensure that, out of all the features selected by a model, the expected proportion of false positives is strictly bounded below a user-defined threshold.

How do we separate true signal from high-dimensional noise when the underlying data distribution is unknown? Researchers have attacked this problem from three distinct angles: synthetic negative controls, rank-ordered optimization penalties, and dynamic error budgets.

Synthetic Decoys as Statistical Calibration

The most prominent framework for controlled feature selection relies on a simple, elegant intuition: if you want to know how often your model mistakenly selects noise, give it a controlled dose of known noise to choose from. This is the foundation of the knockoff framework.

A knockoff feature $\tilde{X}_i$ is a synthetic "digital twin" of an original feature $X_i$. It is carefully constructed to mimic the exact correlation and covariance structure of the original feature set, but it is completely independent of the response variable $Y$ once the true features are accounted for.

Original Features (X)  --->  [ Generative Model ]  --->  Synthetic Knockoffs (X̃)
         |                                                        |
         +-----------------------+--------------------------------+
                                 |
                                 v
                       [ Feature Selection ]
                                 |
         +-----------------------+--------------------------------+
         |                                                        |
         v                                                        v
Selected Originals (Signals)                             Selected Knockoffs (Known Noise)

Because an original feature and its knockoff are statistically exchangeable with respect to the noise, any selection algorithm that cannot tell the difference is equally likely to pick either one. If the model selects a knockoff, it is indisputably a false discovery. By tracking the ratio of selected knockoffs to selected originals, the framework can precisely estimate and control the true FDR (Li et al., Watson).

The core challenge lies in generating these high-quality synthetic twins for complex data.

  • Deep Generative Models: Sesia et al. introduced Deep Knockoffs, utilizing deep generative architectures to construct knockoffs without needing an explicit analytical distribution of the data. Similarly, Jordon et al. developed KnockoffGAN, using generative adversarial networks to handle complex, non-Gaussian feature spaces.
  • Direct Likelihood Optimization: To ensure the synthetic features don't leak information about the response, Sudarshan et al. proposed Deep Direct Likelihood Knockoffs (DDLK), which optimizes the generation process by directly minimizing the KL divergence between original and swapped feature distributions.
  • Scaling and Stability: Generating a single random knockoff can make selection results unstable. To fix this, Roquero Gimenez et al. introduced simultaneous multi-knockoffs, comparing an original feature's importance against the maximum importance of several synthetic controls to lower the detection threshold for sparse signals. For massive datasets, Askari et al. designed FANOK to reduce the computational complexity of knockoff generation from cubic to linear time using low-rank factor models, while Poignard et al. integrated a preliminary screening phase with kernel knockoffs to handle ultra-high dimensions.

Symmetry Without Full Knockoffs

Constructing a perfect joint distribution for knockoffs can be computationally prohibitive. An alternative approach is to inject symmetry directly into the test statistics themselves.

Xing et al. introduced the Gaussian Mirror method. Instead of generating a full twin variable, it creates a mirror version of each predictor by adding random Gaussian perturbations. This forces the resulting test statistics for all irrelevant variables to be perfectly symmetric around zero, allowing researchers to easily isolate and control the false discoveries.

Conversely, Hansen et al. demonstrated with FLOWSELECT that knockoffs can be bypassed entirely by pairing deep density estimators (normalizing flows) with Markov Chain Monte Carlo (MCMC) sampling to compute exact conditional p-values directly.

Baking Error Control into the Loss Function

Instead of generating synthetic data to test a model post-hoc, can we design an optimization objective that inherently controls the FDR during training?

Standard Lasso applies a uniform $\ell_1$ penalty to all regression coefficients. This uniform penalty fails to account for the fact that as you test more variables, the statistical threshold for what constitutes a "true discovery" must become stricter.

Bogdan et al. solved this by introducing SLOPE (Sorted L-One Penalized Estimation). SLOPE replaces the uniform penalty with a rank-ordered penalization sequence. The algorithm sorts the estimated coefficients by their absolute magnitude and applies a progressively larger penalty to the largest coefficients.

$$\text{Penalty} = \sum_{j=1}^p \lambda_j |\beta|_{(j)} \quad \text{where} \quad \lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p$$

This rank-ordered penalty mathematically mirrors the step-down procedures (like the Benjamini-Hochberg correction) used in multiple hypothesis testing. By penalizing the most dominant predictors more heavily when many candidates are present, SLOPE provides built-in FDR control directly through convex optimization. Lee et al. extended this concept to the Dantzig Selector, creating the Ordered Dantzig Selector (ODS) to achieve the same built-in error control with high computational efficiency.

Dynamic Wealth Budgets for Streaming Features

The previous frameworks assume a batch setting where all candidate features are available upfront. But what happens if features arrive sequentially over time in an infinite stream?

Ungar et al. and Dhillon et al. reframed feature selection as a dynamic investment game governed by an Information Investing Criterion (IIC).

[ Start: Total Wealth Budget ]
             |
             v
    ---> [ Test New Feature ] ---> Costs Wealth
   |         |
   |         +---> Is it predictive? 
   |                   |
   |                   +---> YES: Earns Wealth Reward
   |                   |
   |                   +---> NO:  Wealth is Permanently Lost
   |         |
    ---------+---> Adjust Inclusion Threshold Based on Remaining Wealth

The algorithm initializes with a fixed "wealth budget," which represents the total allowable false discovery error. When a new feature arrives from the stream, testing it "costs" a small fraction of wealth. If the feature enters the model and proves to be truly predictive, the model "earns" a wealth reward. If the feature turns out to be spurious noise, the invested wealth is permanently lost.

By dynamically adjusting the statistical threshold for including the next feature based on the remaining wealth, streaming feature selection guarantees that the ratio of spurious to relevant features remains strictly bounded, even across an infinite search space.

Key Foundational Work

Feature Selection using Multiple Streams (2010) — contributed 1 concept elements to this direction.
Streaming Feature Selection using IIC (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_false_discovery_rate_control_high_dimens_5a638aac,
  title = {false discovery rate control (high-dimensional feature selection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/false-discovery-rate-control-high-dimensional-feature-selection/rdr_5a638aac62ae451993bf309fae635d92},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_false_discovery_rate_control_high_dimens_5a638aac_v1,
  title = {false discovery rate control (high-dimensional feature selection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/false-discovery-rate-control-high-dimensional-feature-selection/rdr_5a638aac62ae451993bf309fae635d92/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake