Dynamic Significance Budgets for Streaming Hypothesis Tests

This direction is merged into another direction: anytime-valid inference, dynamic significance budgeting. Its last version stays readable and citable here.

Version 1 of 141 papers63 conceptsfrozen 2026-10-04

In traditional statistics, controlling for false discoveries is a "batch" process. You decide on a set of hypotheses, collect all your data, and apply a correction like Bonferroni or Benjamini-Hochberg. But modern science and industry operate in a stream: tech companies run thousands of A/B tests sequentially, genomic researchers test new variants as they are sequenced, and data scientists explore datasets interactively.

In these "online" settings, the total number of tests is often unknown or even infinite. If you simply keep testing at a fixed significance level (e.g., $\alpha = 0.05$), you will eventually be overwhelmed by false positives. The solution emerging from recent research is to treat statistical significance as a finite, earnable resource: Alpha-Wealth.

The Alpha-Wealth Mechanism

The core intuition behind online multiple testing is to view the allowable error rate as a bank account. This framework, often called $\alpha$-investing, was explored by Zheguang Zhao and colleagues in Controlling False Discoveries During Interactive Data Exploration.

The process follows a simple economic cycle:

  1. Initial Endowment: You start with a small amount of "alpha-wealth" (the total probability of making a Type I error you are willing to risk).
  2. Spending: For every new hypothesis, you "spend" a portion of your wealth by setting a significance threshold for that specific test.
  3. Loss: If the test fails to reject the null hypothesis, that wealth is gone.
  4. Reinvestment: If you make a discovery (reject the null), you "earn" wealth back.

As Adel Javanmard and Andrea Montanari demonstrate in Online Rules for Control of False Discovery Rate and False Discovery Exceedance, this reinvestment is what allows for an infinite stream of tests. Because you only earn budget back when you find a "true" signal, the total number of false discoveries is naturally capped relative to the total number of rejections, maintaining control over the False Discovery Rate (FDR).

Sustainability and the Risk of Alpha-Death

A major challenge in this research is ensuring the system doesn't run out of "money." If a researcher tests a long string of "duds" (null hypotheses), their alpha-wealth will eventually decay to a point where the significance threshold becomes so small that a discovery is mathematically impossible. This state is known as alpha-death.

To combat this, Aaditya Ramdas and Jinjin Tian introduced algorithms like ADDIS and SAFFRON. In SAFFRON: an Adaptive Algorithm for Online Control of the False Discovery Rate, they propose adaptively estimating the proportion of true null hypotheses. By "discarding" p-values that are clearly not discoveries (e.g., $p > 0.5$), the system can reallocate that wasted budget to future tests.

Further refinements, such as the SupLORD framework described in Dynamic Algorithms for Online Multiple Testing, use "non-monotone" spending rules. Instead of a strictly decaying budget, these rules allow the system to hoard wealth during sparse periods and spend it aggressively when discoveries start appearing, significantly increasing the statistical power of the stream.

From A/B Testing to AI Safety

The power of dynamic significance budgeting is its versatility across different domains where decisions are irreversible and sequential:

Why It Works: The Feedback Loop

The elegance of the alpha-wealth approach lies in its self-correcting nature. It transforms a static statistical problem into a dynamic control problem. By rewarding discoveries with the capacity to make more discoveries, the system creates a virtuous cycle for high-signal environments while enforcing a "safety brake" in high-noise environments. As D. S. Robertson notes in Online multiple hypothesis testing, this allows for rigorous, long-term experimentation without ever needing to know when the experiment will end.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_significance_budgets_for_streami_8817c214,
  title = {Dynamic Significance Budgets for Streaming Hypothesis Tests},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-significance-budgets-for-streaming-hypothesis-tests/rdr_8817c21462e94ad2b9a09487024f34b7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_significance_budgets_for_streami_8817c214_v1,
  title = {Dynamic Significance Budgets for Streaming Hypothesis Tests},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-significance-budgets-for-streaming-hypothesis-tests/rdr_8817c21462e94ad2b9a09487024f34b7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}