Online Control of the False Discovery Rate

67 concepts38 papers1 proposal

The Challenge of the Infinite Stream

Traditional statistical inference assumes a static world. If a scientist wants to test 100 genes or evaluate 5 different website layouts, they gather a fixed batch of data, run their tests, and apply a correction like Bonferroni or Benjamini-Hochberg to control the rate of false discoveries. These classical methods require knowing the total number of hypotheses upfront.

But modern data science operates as a continuous, open-ended stream. Consider a tech company running hundreds of A/B tests every week, or an interactive data exploration tool where a user dynamically clicks through charts, implicitly testing hypotheses with every action. As discussed by D. S. Robertson et al., if you apply static corrections to an infinite sequence of tests, your statistical power eventually drops to zero—making it impossible to discover anything new—or your false positive rate explodes.

The Intuition of Alpha-Wealth

To solve this, researchers reframed multiple testing as an economic game. Imagine you are given a bank account containing a finite amount of "alpha-wealth," which represents your total allowable error budget.

When a new hypothesis arrives in the stream, you must "spend" a portion of your alpha-wealth to set its significance threshold ($\alpha_t$). If the test fails to reject the null hypothesis, that wealth is permanently spent. However, if you successfully reject the null hypothesis—making a genuine discovery—you receive a "payout" that replenishes your bank account.

Why does this rule maintain rigorous statistical control? The False Discovery Rate (FDR) is the expected ratio of false discoveries to total discoveries:

$$\text{FDR} = \mathbb{E}\left[ \frac{\text{False Discoveries}}{\max(\text{Total Discoveries}, 1)} \right]$$

Because the error metric is a proportion, every time you make a true discovery, the denominator of this fraction increases. A larger denominator means your global experiment can tolerate more false positives in the future without driving the overall ratio above your target error rate. Therefore, earning a payout upon a successful rejection is not just an arbitrary reward; it is a mathematically justified expansion of your capacity to tolerate future risks.

This wealth-reinvestment paradigm allows for infinite-horizon testing. Early work by Lyle Ungar et al. applied this concept to streaming feature selection, using an information investing criterion to dynamically adjust the threshold for adding new variables to a model without overfitting. As noted by Aaditya Ramdas et al., the key to protecting the budget is being conservative during the initial discovery phase; once a baseline of true discoveries is established, the system can safely award higher significance budgets to subsequent tests.

Preserving Wealth by Discarding Duds

In a real-world stream, many hypotheses are "duds"—true nulls that offer no real signal. If an algorithm blindly spends alpha-wealth testing every unpromising hypothesis, its bank account will quickly hit zero, a failure state known as "alpha-death."

To prevent this, modern online algorithms focus on budget efficiency. The SAFFRON algorithm, introduced by Aaditya Ramdas et al., adaptively estimates the proportion of true null hypotheses in the stream. It establishes a "candidacy" threshold: if an incoming p-value is too large, it is immediately recognized as a non-discovery, and the algorithm avoids wasting any alpha-wealth on it.

Jinjin Tian et al. advanced this intuition with the ADDIS algorithm, which explicitly discards conservative nulls. In practical A/B testing, null distributions are often conservative (skewing toward larger p-values). By intelligently discarding these unpromising tests, the authors demonstrated that you can dramatically increase statistical power without sacrificing error control when the null distributions happen to be perfectly uniform.

Managing Time: Asynchrony and Memory Decay

Deploying alpha-investing in production introduces complex temporal challenges. First, real-world experiments do not finish in a neat, sequential order; they run concurrently and finish asynchronously.

When determining the significance threshold for a new test in an asynchronous stream, you cannot know whether currently running tests will succeed or fail. To prevent over-allocating the budget, Aaditya Ramdas et al. proposed an asynchronous framework that assumes the worst-case outcome for all active, pending tests. By treating unresolved tests as if they failed, the algorithm enforces a temporary conservatism that guarantees the false discovery budget is never overdrawn.

The second challenge is historical bias. If an algorithm accumulates massive alpha-wealth from a string of early successes, it could theoretically authorize a flood of false discoveries later on—a phenomenon known as "piggybacking." Conversely, an early dry spell could permanently bankrupt the system.

To decouple the present from the distant past, Quentin Rebjock et al. and Aaditya Ramdas et al. introduced memory decay mechanisms. By applying a discount factor to past decisions, the system progressively "forgets" older observations. This prevents historical wealth from permanently suppressing or inflating current decision thresholds, ensuring that budget allocation remains tightly coupled to recent data.

Broadening the Framework

The core mechanics of dynamic significance budgeting have been extended to handle structural constraints, side information, and alternative statistical metrics:

  • Contextual Allocation: Rather than treating all incoming hypotheses uniformly, Shiyun Chen et al. utilized neural networks to process unique features or context for each incoming hypothesis, dynamically steering alpha-wealth toward tests with a higher prior probability of success.
  • Batch Processing: When hypotheses arrive in groups rather than a strict sequence, Tijana Zrnic showed that processing tests in batches allows algorithms to adaptively adjust thresholds based on empirical estimates of the false discovery proportion from previous batches, boosting power while maintaining anytime guarantees.
  • Alternative Error Metrics: The framework is not limited to p-values and false discoveries. Asaf Weinstein adapted these principles to control the False Coverage Rate (FCR) for streaming confidence intervals, while Yifan Zhang et al. introduced e-GAI, a framework that leverages e-values and risk-averse wealth allocation to maintain rigorous error control under arbitrary, complex data dependencies.

Key Foundational Work

Streaming Feature Selection using IIC (2005) — contributed 2 concept elements to this direction.
On Efficient Heuristic Ranking of Hypotheses (1997) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_significance_budgeting_d4985939,
  title = {dynamic significance budgeting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-significance-budgeting/rdr_d49859394a5340b0bbf5055252f74d29},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_significance_budgeting_d4985939_v1,
  title = {dynamic significance budgeting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-significance-budgeting/rdr_d49859394a5340b0bbf5055252f74d29/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake