False Discovery Rate Control for High Dimensional Data

146 concepts123 papers3 proposals

The Multiple Testing Crisis

In modern data science, we routinely confront datasets where the number of measured features vastly outnumbers the observations. Whether scanning thousands of genes in a microarray study (John Quackenbush), mapping voxels across fMRI brain scans (Haiyun Ma et al.), or evaluating user segments in large-scale A/B testing (Yuxiang Xie et al.), we face a fundamental statistical hurdle: the multiple testing problem.

If you perform a single hypothesis test at a standard significance level of $\alpha = 0.05$, there is a 5% chance of falsely rejecting a true null hypothesis (a Type I error). However, if you test 10,000 hypotheses simultaneously—such as looking for differential gene expression—pure chance dictates that you will accumulate roughly 500 false positives.

Historically, statisticians controlled the Family-Wise Error Rate (FWER), which guarantees the probability of committing at least one false positive remains below $\alpha$. The classic Bonferroni correction achieves this by dividing $\alpha$ by the total number of tests. But as Denis Chetverikov et al. note, such univariate corrections become excessively conservative in high-dimensional settings. By forcing the threshold to be so strict, FWER corrections destroy statistical power, causing researchers to miss genuine biological or physical signals.

False Discovery Rate (FDR) control re-frames the objective. Instead of eliminating false positives entirely, FDR control bounds the expected proportion of false discoveries among all declared discoveries. If an algorithm yields 100 significant hits at an FDR of 5%, you can proceed knowing that roughly 5 of those hits are likely false positives, while the remaining 95 are true signals. This compromise dramatically boosts discovery power while maintaining rigorous error boundaries.

The Intuition of Adaptive Thresholding

The foundational mechanism for controlling FDR is the Benjamini-Hochberg (BH) step-up procedure, utilized across diverse fields from clinical cohort studies (Robert J. Smith et al.) to transcriptomics (Davina Derous et al.). To understand why it works, imagine sorting your $m$ calculated p-values from smallest to largest:

$$p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}$$

Instead of comparing every p-value to a fixed threshold $\alpha$, the BH procedure compares each $p_{(i)}$ to a dynamically scaling threshold that depends on its rank $i$:

$$\text{Threshold}(i) = \frac{i}{m} \alpha$$

The algorithm steps up from the largest p-value to the smallest, finding the largest rank $k$ where the condition is met:

$$k = \max \left\{ i : p_{(i)} \le \frac{i}{m} \alpha \right\}$$

It then rejects all null hypotheses up to rank $k$.

p-value
  ^
  |                                       .  (Uniform Null Distribution)
  |                                    .  /
  |                                 .    /  Linear BH Threshold (i/m * alpha)
  |                              .      /
  |                           .        /
  |                        *          /
  |                  *   /           .
  |            *       /            .
  |      *           /             .
  |  *             /              .
  +-----------------------------------------> Rank (i)
     [ Rejection Region ]

The intuition behind this adaptive threshold is elegant. Under the null hypothesis, p-values are uniformly distributed between 0 and 1, forming a straight diagonal line when plotted against their rank. True discoveries deviate from this line, clustering near zero. By scaling the threshold linearly with the rank $i$, the BH procedure adaptively relaxes or tightens its stringency based on the density of evidence. If there are many strong signals, $k$ becomes large, elevating the allowable p-value threshold and preserving statistical power. If no true signals exist, the threshold drops toward the conservative Bonferroni limit ($\frac{\alpha}{m}$), protecting against false discoveries.

As Lorenzo De Stefani et al. demonstrate, this adaptive balance is vital for modern interactive data exploration tools, where automated recommendations can easily act as "p-hacking machines" without real-time FDR tracking.

Beyond Flat Hypotheses: Hierarchical and Online Contexts

The classical BH procedure treats all hypotheses as a flat, static collection. However, real-world scientific questions often possess intrinsic structure or arrive sequentially over time.

Hierarchical FDR

In feature interpretation and clinical diagnostics, hypotheses are frequently nested. For instance, when interpreting machine learning models or analyzing clinical variables, features can be grouped into broader taxonomic categories. Kyubin Lee et al. and Anat Reiner‐Benaim et al. leverage hierarchical FDR procedures to handle this structure. By testing higher-level groups before drilling down into specific sub-features, these algorithms mitigate the risk of false positives at multiple resolutions without sacrificing the granularity of the final discoveries.

Online FDR

In continuous A/B testing or streaming data analysis, hypotheses arrive one by one over time, and decisions must be made instantaneously. Standard batch FDR control fails here because the total number of tests $m$ is unknown or infinite. To maintain control over an infinite horizon, online algorithms view FDR control as managing an "error wealth" budget.

The ADDIS algorithm, introduced by Jinjin Tian et al., addresses a common real-world problem: conservative nulls, where null hypotheses generate p-values stochastically larger than a uniform distribution. ADDIS adaptively discards p-values that fall above a certain threshold, recognizing early on that they are highly unlikely to be discoveries. By ignoring these uninformative hypotheses, the algorithm avoids wasting its error budget, thereby maintaining high statistical power for future streaming tests.

Powering Discoveries with Auxiliary Covariates

Traditional FDR methods treat every hypothesis as an isolated entity with an equal prior probability of being true. Yet, researchers often possess rich contextual metadata—or auxiliary covariates—about each test. For example, in genomic scans, a genetic variant's physical location or its evolutionary conservation score might indicate its likelihood of being truly associated with a disease.

To exploit this hidden information, Lin Qiu et al. developed NeurT-FDR, a probabilistic framework that pairs deep neural networks with FDR control. The neural network ingests auxiliary covariates to learn a prior weight for each hypothesis. Tests that are contextually more likely to yield true discoveries are assigned a higher weight, effectively lowering their p-value threshold. Conversely, hypotheses with unpromising covariates face a stricter penalty. This covariate-powered approach extracts significantly more true discoveries from high-dimensional datasets while mathematically guaranteeing that the overall FDR remains bounded.

Empirical Estimation and Robustness

When dealing with heavy-tailed data or complex, unknown correlations between variables, analytical formulas for calculating p-values often break down. To prevent structural assumptions from undermining error control, modern frameworks rely on empirical estimation.

Robust Estimators

When data deviates significantly from standard Gaussian assumptions, standard parametric tests fail. To address this, Jianqing Fan et al. integrated robust, rank-based estimators with error-control algorithms. This combination ensures that even when data is "messy" or heavy-tailed, the calculated p-values remain valid inputs for the FDR control pipeline.

Resampling and Negative Controls

Instead of assuming a theoretical null distribution, empirical strategies simulate the null distribution directly from the data. Hebing Chen et al. utilize bootstrap methods to empirically estimate q-values—the FDR equivalent of a p-value for a given hypothesis.

In high-throughput biological screenings, such as pooled CRISPR screens, researchers embed physical "negative controls" directly into the experiment. As Sunil Bodapati et al. demonstrate, tracking how often these non-targeting negative control guides are falsely flagged as significant allows for a direct, empirical calibration of the false discovery rate. This empirical feedback loop separates genuine biological signals from background noise and off-target effects, ensuring that high-dimensional discoveries can be reliably replicated in follow-up validation studies.

Key Foundational Work

Controlling the False-Discovery Rate in Astrophysical Data Analysis (2001) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_false_discovery_rate_control_false_disco_83af5f85,
  title = {false discovery rate control (false discovery rate correction)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/false-discovery-rate-control-false-discovery-rate-correction/rdr_83af5f859f1744008a476ef0e09e48af},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_false_discovery_rate_control_false_disco_83af5f85_v1,
  title = {false discovery rate control (false discovery rate correction)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/false-discovery-rate-control-false-discovery-rate-correction/rdr_83af5f859f1744008a476ef0e09e48af/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake