Detecting Multiple Replicating Signals using Adaptive Filtering Procedures

Jingshu Wang, Lin Gui, Weijie Su, Chiara Sabatti, Art B. Owen

Introduction

In the wake of the "reproducibility crisis," scientific focus has shifted toward identifying "replicating signals"—findings that appear consistently across multiple independent studies. This is formally addressed through Partial Conjunction (PC) testing, which asks whether a signal is independently significant in at least $r$ out of $n$ studies.

However, in high-throughput fields like genomics, researchers often test millions of hypotheses (e.g., genes or SNPs) simultaneously. To avoid false positives, they must apply multiplicity corrections like the Bonferroni or Benjamini-Hochberg procedures. Because PC null hypotheses are "composite," standard corrections become extremely conservative. In sparse settings where only a few signals are real, the "statistical tax" of testing millions of items often crushes the actual signal, leading to zero discoveries even when replicable effects exist.

The AdaFilter Framework

The authors introduce AdaFilter, an adaptive filtering procedure that "learns" from the data to reduce the multiplicity burden. The core logic of AdaFilter involves calculating two distinct values for every hypothesis:

  1. Filtering p-value ($F_j$): Derived from the $(r-1)$-th smallest p-value among $n$ studies, used to decide if a hypothesis is a candidate for replication.
  2. Selection p-value ($S_j$): Derived from the $r$-th smallest p-value, used for the actual rejection decision.

Unlike standard methods that treat every hypothesis as equally likely to be null, AdaFilter adaptively sets a threshold. If many hypotheses appear to be "obvious nulls" based on their filtering p-values, they are effectively ignored. This reduces the total number of tests being counted against the significance budget, thereby increasing the power to detect the remaining candidates. Crucially, the authors provide rigorous proofs that this data-driven approach maintains control over the Family-Wise Error Rate (FWER) and False Discovery Rate (FDR).

Key Insights

The Multiplicity Tax is Flexible

The most striking insight of this work is that researchers do not have to pay the full multiplicity tax for every hypothesis. By identifying which hypotheses are clearly not going to meet the replication threshold $r$, statistical power can be "saved" for those that might.

Lack of "Complete Monotonicity"

In a departure from common statistical properties, AdaFilter violates "complete monotonicity." Usually, making a p-value smaller only helps a discovery. In AdaFilter, making one gene's p-value smaller might actually make it harder for a different gene to be rejected because it changes the adaptive threshold. The authors argue this "instability" is actually the source of the method's superior power.

Computational Efficiency

Existing Bayesian methods, such as repfdr, are powerful but can become exponentially expensive as the number of studies $n$ grows. AdaFilter is a frequentist approach with a computational cost of only $O(Mn \log n)$, making it feasible for massive modern datasets.

Experimental Results

Simulations

In synthetic datasets mimicking gene expression and genetic regulation, AdaFilter consistently demonstrated much higher recall than standard Bonferroni or BH adjustments, particularly when the signal was sparse.

Duchenne Muscular Dystrophy (DMD) Microarrays

The authors analyzed four independent datasets from the NCBI Gene Expression Omnibus involving muscle tissue samples. When looking for genes that replicate across all four studies ($r=4$), standard Fisher-based methods found only 2 significant genes. AdaFilter BH identified 32, including several well-known marker genes for muscle contraction that other methods missed.

Metabolomics GWAS

In a massive Genome-Wide Association Study (GWAS) involving 2.1 million SNPs and 333 metabolite traits, AdaFilter identified significantly more SNPs associated with metabolic pathways than the empirical Bayes method repfdr, with the performance gap widening as the requirement for replication ($r$) increased.

Limitations and Future Work

Currently, the theoretical proof for error control assumes that p-values are independent across different studies and hypotheses. While simulations suggest AdaFilter is robust to "within-study" dependence (like gene correlation), a formal proof remains an open challenge. Additionally, the replication level $r$ must be pre-specified; the current version of AdaFilter is not yet designed for "fishing" for the best $r$ across a range of values, as that would introduce additional multiplicity issues.

Summary updated

Abstract

Replicability is a fundamental quality of scientific discoveries: we are interested in those signals that are detectable in different laboratories, study populations, across time etc. Unlike meta-analysis which accounts for experimental variability but does not guarantee replicability, testing a partial conjunction (PC) null aims specifically to identify the signals that are discovered in multiple studies. In many contemporary applications, e.g., comparing multiple high-throughput genetic experiments, a large number $M$ of PC nulls need to be tested simultaneously, calling for a multiple comparisons correction. However, standard multiple testing adjustments on the $M$ PC $p$-values can be severely conservative, especially when $M$ is large and the signals are sparse. We introduce AdaFilter, a new multiple testing procedure that increases power by adaptively filtering out unlikely candidates of PC nulls. We prove that AdaFilter can control FWER and FDR as long as data across studies are independent, and has much higher power than other existing methods. We illustrate the application of AdaFilter with three examples: microarray studies of Duchenne muscular dystrophy, single-cell RNA sequencing of T cells in lung cancer tumors and GWAS for metabolomics.
BibTeX
@article{wang2022detecting,
  author = {Jingshu Wang and Lin Gui and Weijie Su and Chiara Sabatti and Art B. Owen},
  title = {{Detecting Multiple Replicating Signals using Adaptive Filtering Procedures}},
  journal = {The Annals of Statistics},
  year = {2022},
  doi = {10.1214/21-aos2139},
}
Figure