Dependence Robust False Discovery Rate Control

277 concepts146 papers1 proposal

In large-scale statistical inference, we rarely test a single hypothesis in isolation. Whether analyzing thousands of genes in a microarray experiment, scanning voxels in a brain scan, or running continuous A/B tests on a web platform, we face the multiple testing problem: as the number of tests grows, the probability of encountering false positives explodes.

To combat this, statisticians typically control the False Discovery Rate (FDR)—the expected proportion of false discoveries among all rejected hypotheses. However, classical FDR-controlling procedures, such as the seminal Benjamini-Hochberg (BH) procedure, rely heavily on the assumption that the underlying test statistics are independent or exhibit highly specific forms of positive dependence.

In real-world datasets, this independence assumption is routinely violated. Genes operate in highly correlated pathways; neighboring voxels in brain imaging share structural noise; and sequential data streams exhibit temporal dependencies. When faced with complex, unknown, or adversarial dependencies, standard error-control frameworks can fail catastrophically, leading to a massive inflation of false positives or forcing researchers into overly conservative corrections that destroy the statistical power to find genuine discoveries.

Modern research addresses this challenge through three main paradigms: shifting from probability-based evidence (p-values) to expectation-based evidence (e-values), using data-driven resampling to capture empirical covariance, and designing dependency-aware online streaming algorithms.

Why P-Values Fail Under Dependency

To understand why dependency breaks traditional multiple testing, we must look at how p-values behave. A p-value is a probability-based measure of evidence. Under the true null hypothesis, a valid p-value is stochastically larger than or equal to a uniform distribution:

$$P(P_i \le t) \le t \quad \forall t \in [0, 1]$$

When controlling the FDR across $m$ hypotheses, an algorithm must evaluate the joint behavior of these p-values. If the tests are independent, the number of null p-values falling below a given threshold follows a predictable binomial distribution.

However, if the hypotheses are arbitrarily dependent, their joint distribution becomes highly unpredictable. As Chakravarthi Kanduri demonstrates in Beware of counter-intuitive levels of false discoveries in datasets with strong intra-correlations, strong intra-correlations can cause standard FDR controls to emit thousands of false positives even when absolutely no real biological signal exists.

To guarantee error control under any possible dependency structure using p-values, statisticians must apply worst-case mathematical corrections, such as the Benjamini-Yekutieli (BY) procedure. The BY correction penalizes the significance threshold by a logarithmic factor $\sum_{i=1}^m (1/i) \approx \log(m)$. In high-dimensional data where $m$ is in the thousands or millions, this penalty severely reduces the probability of making true discoveries, creating a harsh trade-off between robustness and sensitivity.

E-Values: Trading Probabilities for Expectations

The fundamental reason p-values require a logarithmic penalty under arbitrary dependence is that probabilities do not aggregate linearly when variables are correlated. To bypass this limitation, a rapidly growing body of work replaces p-values with e-values.

An e-value is a non-negative random variable $E$ that measures evidence against the null hypothesis, calibrated such that its expected value under the null is bounded by 1:

$$\mathbb{E}[E] \le 1$$

The mathematical elegance of e-values lies in the linearity of expectation. Unlike probabilities, the expectation of a sum of random variables is always the sum of their expectations, regardless of how they are correlated:

$$\mathbb{E}\left[\sum_{i=1}^m E_i\right] = \sum_{i=1}^m \mathbb{E}[E_i]$$

Because expectations naturally ignore joint distribution geometries, e-value-based multiple testing procedures can control the FDR under completely arbitrary and unknown dependency structures without requiring any mathematical penalties. As explained by Aaditya Ramdas et al. in Hypothesis testing with e-values, this expectation-based framing provides a robust alternative to traditional corrections, maintaining mathematical validity where p-value guarantees collapse.

P-Values (Probability-Based)          E-Values (Expectation-Based)
  P(P_i ≤ t) ≤ t                         E[E_i] ≤ 1
  Aggregating requires joint             Aggregating relies on linearity
  probabilities (highly sensitive        of expectation (immune to
  to correlation structures).            correlation structures).
  Result: Requires log(m) penalty        Result: No penalty required
  under arbitrary dependence.            under arbitrary dependence.

This structural advantage makes e-values highly effective in complex, adaptive testing environments. For instance, Ziyu Xu et al. introduce a martingale-based framework using e-processes in A unified framework for bandit multiple testing. Their approach allows multiple agents to explore data and stop adaptively while maintaining robust FDR control under complex dependencies, completely eliminating the logarithmic corrections that would typically plague a p-value method. Similarly, Yifan Zhang et al. develop e-GAI: e-value-based Generalized $\alpha$-Investing for Online False Discovery Rate Control, demonstrating that e-values allow streaming discovery algorithms to remain reliable in time-series and sensor monitoring applications where data points exhibit severe, unknown correlations.

Data-Driven Resampling: Capturing Empirical Covariance

When researchers must use traditional p-values or test statistics, an alternative to assuming worst-case dependencies is to empirically estimate the exact correlation structure directly from the data. Instead of relying on rigid parametric models, these methods use bootstrap and resampling techniques to learn how the tests co-vary.

In high-dimensional settings where the number of features far exceeds the sample size, standard meta-analysis combinations (like Fisher's or Stouffer's tokens) inflate false positives because they treat correlated tests as independent pieces of evidence. To resolve this, Xiufan Yu et al. propose a framework in A Unified Combination Framework for Dependent Tests with Applications to Microbiome Association Studies that employs a parametric bootstrap to empirically estimate the joint distribution of test statistics under the null hypothesis. This adjustment explicitly maps the mathematical dependencies, preventing false-positive inflation during feature aggregation.

In mass-univariate linear modeling, such as neuroimaging, Bertrand Thirion et al. demonstrate in FDP control in mass-univariate linear models using the residual bootstrap that data-driven resampling provides significantly tighter bounds on the false discovery proportion (FDP) than parametric alternatives. By using a residual bootstrap, the method captures the complex spatial noise inherent in biological data, avoiding the over-conservatism of methods that must protect against all theoretical dependence scenarios.

Similarly, for high-dimensional genomic data with multiple heterogeneous outcomes, Jin-Hong Du et al. implement a Gaussian multiplier bootstrap in Causal Inference for Genomic Data with Multiple Heterogeneous Outcomes. This mechanism captures the underlying correlation structure across thousands of outcomes, allowing the multiple testing procedure to maintain strict error thresholds without sacrificing the statistical power needed to detect true causal relationships.

Online Error Control in Correlated Data Streams

The challenge of dependency becomes even more pronounced in "online" multiple testing, where hypotheses arrive sequentially in a stream, and the researcher must make an immediate, irrevocable decision to reject or accept each hypothesis without knowing how many tests will follow.

Early online algorithms, such as basic $\alpha$-investing or early iterations of the LORD and SAFFRON algorithms, assumed that the streaming p-values were independent. As noted by Aaditya Ramdas et al. in SAFFRON: an Adaptive Algorithm for Online Control of the False Discovery Rate and Online control of the false discovery rate with decaying memory, extending online error control to handle complex, high-dimensional dependencies remains a critical hurdle for real-world data streams.

To maintain a stable FDR in streaming environments with dependent data, modern algorithms dynamically adjust their significance thresholds using specialized mathematical "shape functions" or calibration adjustments.

For instance, Aaron J. Fisher introduces the TOAD framework in Online Control of the False Discovery Rate under "Decision Deadlines". TOAD allows researchers to update their decisions up until a specific deadline, employing specific threshold-shaping functions that guarantee strict FDR control even when the streaming data points exhibit arbitrary or positive dependencies.

[Hypothesis Stream] ──> [Dependency-Aware Filter] ──> [Dynamic Threshold Adjustment]
                             │                                     │
                             ├──> Shared Calibration Set           └──> Shape Functions /
                             └──> Temporal/Local Correlations           Risk Aversion

This strategy is also vital for modern machine learning tasks like out-of-distribution (OOD) detection, where a model must continuously determine if incoming data samples belong to the training distribution. In An Online Statistical Framework for Out-of-Distribution Detection, Xinsong Ma et al. present the g-LOND algorithm. Because streaming OOD detectors typically calculate p-values using a shared, static calibration dataset, the resulting p-values are inherently dependent. The authors overcome this by dynamically adjusting significance levels using function classes that explicitly account for these calibration-induced dependencies, ensuring that the false alarm rate remains strictly bounded over time.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 168 papers
  2. Version 1 · 2026-10-04 · 158 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_dependence_robust_error_control_6920c59d,
  title = {dependence-robust error control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dependence-robust-error-control/rdr_6920c59d29f74e31acf1efa9670c9c21},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_dependence_robust_error_control_6920c59d_v2,
  title = {dependence-robust error control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dependence-robust-error-control/rdr_6920c59d29f74e31acf1efa9670c9c21/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake