Multiple Testing for Dependent and Sequential Data

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2158 papers192 conceptsfrozen 2026-10-04

In modern data science, we rarely test a single hypothesis in isolation. Whether it is a tech company running thousands of A/B tests on website features or a genomic researcher scanning millions of genetic variants for disease associations, we face the "multiple testing" problem: the more questions you ask, the more likely you are to get a "yes" just by sheer luck.

Traditional solutions, like the Benjamini-Hochberg procedure, were designed for an "offline" world where all test results are available at once. However, as noted by Adel Javanmard and Andrea Montanari in On Online Control of False Discovery Rate, modern science often involves a continuous stream of hypotheses arriving over months or years. This shift from batch processing to sequential streams introduces two fundamental challenges: maintaining statistical validity over time and handling the complex dependencies between tests.

The Online Frontier: Managing Alpha Wealth

The core intuition behind online error control is the concept of an "alpha-wealth" budget. Imagine you have a total budget for false positives (the significance level $\alpha$). Every time you run a test, you "spend" a bit of this wealth. If a test results in a discovery, you "earn" back some wealth to spend on future tests.

Algorithms like SAFFRON, developed by Aaditya Ramdas, Tijana Zrnic, and colleagues in SAFFRON: an Adaptive Algorithm for Online Control of the False Discovery Rate, improve this process by being adaptive. Instead of treating every test as equally likely to be a discovery, SAFFRON estimates the proportion of "true nulls" (duds). By identifying tests that are likely to lead nowhere, the algorithm can reallocate its budget to more promising hypotheses, significantly increasing discovery power.

This becomes more complex in asynchronous environments. Tijana Zrnic, Michael I. Jordan, and others explored this in Asynchronous Online Testing of Multiple Hypotheses, where experiments overlap in time. In these cases, the decision for one test might need to be made before the results of previous tests are even known, requiring a careful "discounting" of the error budget to account for the lag.

The Dependency Gap

The "Achilles' heel" of multiple testing theory is the assumption of independence. Most mathematical guarantees for False Discovery Rate (FDR) control assume that the result of one test tells you nothing about the next. In reality, this is almost never true:

  • Biological Correlation: In microbiome studies, as Xiufan Yu and Lingzhou Xue point out in A Unified Combination Framework for Dependent Tests, different species of bacteria are part of the same ecosystem, meaning their presence is naturally correlated.
  • Overlapping Data: In A/B testing, different experiments might run on the same user base, creating "local dependencies" that can inflate false positives if ignored.
  • Sequential Cascades: In machine learning, Adam Fisch and Regina Barzilay show in Efficient Conformal Prediction via Cascaded Inference that using a sequence of models to prune candidates creates highly correlated test statistics.

When tests are dependent, standard corrections become either too "loose" (allowing too many false positives) or too "conservative" (missing real signals). For example, Małgorzata Bogdan and Emmanuel Candes noted in their work on SLOPE that while FDR control is rigorous for independent predictors, correlated variables require heuristic adjustments to maintain the same level of integrity.

From P-values to E-values: A Game-Theoretic Solution

To solve the problems of "p-hacking" and data-peeking, a new framework has emerged: Safe Anytime-Valid Inference (SAVI). As detailed by Glenn Shafer, Peter Grünwald, and Vladimir Vovk in Game-theoretic statistics and safe anytime-valid inference, this approach replaces traditional p-values with e-values (or "betting scores").

In this framework, testing is viewed as a game. You start with \$1. If the null hypothesis is true, it is impossible to grow your wealth indefinitely by betting on outcomes. If you manage to turn your \$1 into \$100, you have strong evidence against the null. The beauty of e-values is that they are "anytime-valid": you can stop the experiment whenever you want, or even keep it running forever, and the statistical guarantee remains intact.

This game-theoretic approach is now being integrated into online testing. Lasse Fischer recently proposed Online ARC procedures, which allow researchers to change their minds—rejecting a hypothesis they previously accepted as more evidence arrives—while still maintaining rigorous error control under arbitrary stopping times.

Why This Matters

The shift toward adaptive, online, and dependency-robust testing represents a move toward "real-world" statistics. By moving away from the rigid requirement of independence and the "one-shot" nature of traditional tests, these methods allow scientists to:

  1. Iterate Faster: Make decisions on the fly without waiting for a massive batch of data to finish.
  2. Use Side Information: Leverage covariates and prior knowledge to boost discovery power (as seen in AdaFDR by James Y. Zou).
  3. Maintain Integrity: Ensure that "discoveries" in complex, correlated datasets like the human genome or the brain (see residual bootstrap methods) are actually real.

The ultimate goal of this research direction is a unified framework where statistical significance is not a static threshold, but a dynamic, robust measure that survives the complexities of modern, interconnected data.

Papers in this version

Showing 60 of 158 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 168 papers
  2. Version 1 · 2026-10-04 · 158 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_dependence_robust_error_control_6920c59d,
  title = {dependence-robust error control},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dependence-robust-error-control/rdr_6920c59d29f74e31acf1efa9670c9c21},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_multiple_testing_for_dependent_and_seque_6920c59d_v1,
  title = {Multiple Testing for Dependent and Sequential Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multiple-testing-for-dependent-and-sequential-data/rdr_6920c59d29f74e31acf1efa9670c9c21/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}