Introduction
In the wake of the "reproducibility crisis," scientific focus has shifted toward identifying "replicating signals"—findings that appear consistently across multiple independent studies. This is formally addressed through Partial Conjunction (PC) testing, which asks whether a signal is independently significant in at least $r$ out of $n$ studies.
However, in high-throughput fields like genomics, researchers often test millions of hypotheses (e.g., genes or SNPs) simultaneously. To avoid false positives, they must apply multiplicity corrections like the Bonferroni or Benjamini-Hochberg procedures. Because PC null hypotheses are "composite," standard corrections become extremely conservative. In sparse settings where only a few signals are real, the "statistical tax" of testing millions of items often crushes the actual signal, leading to zero discoveries even when replicable effects exist.
The AdaFilter Framework
The authors introduce AdaFilter, an adaptive filtering procedure that "learns" from the data to reduce the multiplicity burden. The core logic of AdaFilter involves calculating two distinct values for every hypothesis:
- Filtering p-value ($F_j$): Derived from the $(r-1)$-th smallest p-value among $n$ studies, used to decide if a hypothesis is a candidate for replication.
- Selection p-value ($S_j$): Derived from the $r$-th smallest p-value, used for the actual rejection decision.
Unlike standard methods that treat every hypothesis as equally likely to be null, AdaFilter adaptively sets a threshold. If many hypotheses appear to be "obvious nulls" based on their filtering p-values, they are effectively ignored. This reduces the total number of tests being counted against the significance budget, thereby increasing the power to detect the remaining candidates. Crucially, the authors provide rigorous proofs that this data-driven approach maintains control over the Family-Wise Error Rate (FWER) and False Discovery Rate (FDR).
Key Insights
The Multiplicity Tax is Flexible
The most striking insight of this work is that researchers do not have to pay the full multiplicity tax for every hypothesis. By identifying which hypotheses are clearly not going to meet the replication threshold $r$, statistical power can be "saved" for those that might.
Lack of "Complete Monotonicity"
In a departure from common statistical properties, AdaFilter violates "complete monotonicity." Usually, making a p-value smaller only helps a discovery. In AdaFilter, making one gene's p-value smaller might actually make it harder for a different gene to be rejected because it changes the adaptive threshold. The authors argue this "instability" is actually the source of the method's superior power.
Computational Efficiency
Existing Bayesian methods, such as repfdr, are powerful but can become exponentially expensive as the number of studies $n$ grows. AdaFilter is a frequentist approach with a computational cost of only $O(Mn \log n)$, making it feasible for massive modern datasets.
Experimental Results
Simulations
In synthetic datasets mimicking gene expression and genetic regulation, AdaFilter consistently demonstrated much higher recall than standard Bonferroni or BH adjustments, particularly when the signal was sparse.
Duchenne Muscular Dystrophy (DMD) Microarrays
The authors analyzed four independent datasets from the NCBI Gene Expression Omnibus involving muscle tissue samples. When looking for genes that replicate across all four studies ($r=4$), standard Fisher-based methods found only 2 significant genes. AdaFilter BH identified 32, including several well-known marker genes for muscle contraction that other methods missed.
Metabolomics GWAS
In a massive Genome-Wide Association Study (GWAS) involving 2.1 million SNPs and 333 metabolite traits, AdaFilter identified significantly more SNPs associated with metabolic pathways than the empirical Bayes method repfdr, with the performance gap widening as the requirement for replication ($r$) increased.
Limitations and Future Work
Currently, the theoretical proof for error control assumes that p-values are independent across different studies and hypotheses. While simulations suggest AdaFilter is robust to "within-study" dependence (like gene correlation), a formal proof remains an open challenge. Additionally, the replication level $r$ must be pre-specified; the current version of AdaFilter is not yet designed for "fishing" for the best $r$ across a range of values, as that would introduce additional multiplicity issues.
Summary updated