Abstract
We propose a method for multiple hypothesis testing with familywise error rate (FWER) control, called the i-FWER test. Most testing methods are predefined algorithms that do not allow modifications after observing the data. However, in practice, analysts tend to choose a promising algorithm after observing the data; unfortunately, this violates the validity of the conclusion. The i-FWER test allows much flexibility: a human (or a computer program acting on the human's behalf) may adaptively guide the algorithm in a data-dependent manner. We prove that our test controls FWER if the analysts adhere to a particular protocol of "masking" and "unmasking". We demonstrate via numerical experiments the power of our test under structured non-nulls, and then explore new forms of masking.
Introduction
Multiple hypothesis testing is a cornerstone of modern scientific discovery, particularly in fields like genomics and neuroimaging where thousands of tests are conducted simultaneously. To prevent spurious findings, researchers typically control the Familywise Error Rate (FWER)—the probability of making at least one false discovery. However, standard FWER-controlling procedures, such as the Bonferroni or Šidák corrections, are rigid; they require the entire testing algorithm to be fixed before data collection.
In practice, data is often messy, and analysts frequently observe patterns that suggest a pivot in strategy. Normally, this "peeking" leads to biased results and contributes to the reproducibility crisis. The i-FWER test addresses this fundamental tension between statistical validity and analyst flexibility, allowing researchers to use intuition and data-driven insights without "cheating."
The i-FWER Protocol
The i-FWER test bridges the gap between rigor and flexibility through a "masking and unmasking" protocol. This framework allows for data-dependent modifications while strictly maintaining FWER control.
The process involves four key steps:
- Masking: Each p-value ($P_i$) is decomposed into two independent parts: a "masked" value $g(P_i)$ that provides a hint about the hypothesis's potential as a discovery, and a "missing bit" $h(P_i)$ used strictly for error estimation.
- Interaction: The analyst (or an automated agent) examines the masked values $g(P_i)$ alongside side information, such as spatial coordinates or prior knowledge, to prioritize hypotheses.
- Unmasking: As hypotheses are iteratively excluded from the potential rejection set, their missing bits $h(P_i)$ are revealed. This allows the analyst to update an empirical estimate of the FWER.
- Stopping: The process continues until the estimated FWER drops below a target threshold $\alpha$. Only then are the remaining hypotheses in the rejection set $\mathcal{R}_t$ officially reported.
Key Insights
The mathematical foundation of i-FWER lies in the independence of $g(P)$ and $h(P)$ under the null hypothesis. This ensures that any selection bias introduced by looking at $g(P)$ to make decisions does not affect the validity of the error estimate derived from $h(P)$.
Unlike traditional "black box" corrections, i-FWER allows for a "human-in-the-loop" approach. Analysts can use subjective expertise or complex machine learning models (like EM algorithms) to guide the search for discoveries. Furthermore, the authors demonstrate that if null p-values are conservative (biased toward 1), specific masking functions like the "Railway" function can exploit this to increase the test's power.
Empirical Performance
The paper evaluates the i-FWER test using synthetic grid data, hierarchical tree data, and real-world RNA-Seq genetic data (the "Airway" dataset). In a 30x30 grid simulation where non-nulls form a spatial cluster, i-FWER significantly outperformed traditional methods.
As shown in the figure above, i-FWER's ability to leverage spatial information leads to a much higher discovery rate (18 vs 7 rejections) while maintaining strict error control. On the "Airway" dataset, which contains over 33,000 genes, the i-FWER test identified 1,849 differentially expressed genes compared to 1,645 identified by Independent Hypothesis Weighting (IHW) at $\alpha=0.2$.
Discussion and Future Work
While i-FWER offers unprecedented flexibility, it currently assumes that null p-values are mutually independent. While sensitivity analyses for correlated data are promising, formal extensions to general dependency structures remain an open challenge. Additionally, the choice of the masking parameter $p_*$ affects power; while $p_* \approx \alpha/2$ is a robust default, the optimal choice depends on the unknown signal strength. The i-FWER test represents a significant step toward adaptive scientific discovery, providing a robust framework for "peeking without cheating."
Summary updated