In the world of statistics, "data snooping" is traditionally viewed as a cardinal sin. Standard multiple testing procedures—like the Bonferroni correction or the Benjamini-Hochberg (BH) method—require researchers to commit to a specific rejection rule before looking at the data. If you peek at the results, tune parameters, and then decide which hypotheses to reject, you have invalidated your statistical guarantees.
This paper, a book chapter by Blanchard, Neuvial, and Roquain, surveys a powerful alternative: post hoc (or "user-agnostic") bounds. These methods provide a mathematical safety net that remains valid no matter how much a researcher explores, slices, or dices the data after the fact.
The Challenge: The Selection Effect
The core issue is the selection effect. In practice, scientific discovery is rarely a rigid, pre-specified pipeline. Researchers often use exploratory tools like volcano plots to identify interesting clusters of genes and then focus their analysis on those specific groups.
Standard methods cannot handle this. If you select a subset $S$ because it looks "promising" in the data, the $p$-values in that subset are no longer representative. This "cherry-picking" leads to an uncontrolled number of false positives. The authors' framework provides a function $V(S)$ that acts as a $(1-\alpha)$-confidence bound on the number of false positives in any subset $S$, valid uniformly across all possible subsets the user might choose.
Methodology: Templates and Calibration
The paper builds a hierarchy of approaches to solve this problem:
- Classical Post Hoc Bounds: The Simes bound offers a starting point but relies on specific dependence assumptions (PRDS) that are often violated in complex biological data.
- Template-based Bounds: The authors introduce "templates"—families of curves that define a spectrum of possible thresholds.
- $\lambda$-Calibration via Permutation: Instead of assuming independence, the authors use permutation tests (randomly shuffling labels in the data) to calibrate the threshold. This "learns" the dependence structure of the data, making the bounds much sharper and more reliable.
- Reference Families: By defining a collection of "reference sets" (e.g., genes grouped by chromosome), researchers can bound false positives within these sets and interpolate a bound for any arbitrary subset $S$ chosen by the user.
Key Results and Insights
The authors demonstrated their approach using the ALL Microarray Data, a genomic dataset involving B-cell acute lymphoblastic leukemia.
- Sharper Confidence Envelopes: The permutation-calibrated bounds significantly outperformed traditional methods. While the Simes bound suggested a False Discovery Proportion (FDP) of less than 0.48 for the top 163 genes, the calibrated linear bound slashed this to 0.23.
- Beta Templates Win: For researchers seeking a small number of highly significant results, "Beta templates" (based on the distribution of order statistics) provided much tighter bounds than standard linear templates.
- Validating Exploratory Selections: The framework rigorously bounded false positives in subsets selected based on both $p$-values and "fold change" (a common but statistically messy practice). For a set of 151 genes selected this way, they could guarantee at least 115 were true discoveries.
Conclusion
While powerful, the approach relies on the Randomization (Rand) assumption, which may not hold in all experimental designs. Additionally, calculating optimal bounds is computationally intensive, requiring efficient approximations. However, by accounting for correlations through permutation and providing flexible templates, this framework transforms "data snooping" into a rigorous, valid tool for exploratory data analysis.
Summary updated