Detecting multiple replicating signals using adaptive filtering procedures

Jingshu Wang, Lin Gui, Weijie Su, Chiara Sabatti, Art B. Owen

TL;DR

This paper introduces AdaFilter, a novel statistical procedure designed to boost the power of detecting "replicating signals" across multiple scientific studies. By adaptively filtering out less likely null hypotheses, AdaFilter significantly increases the ability to identify true findings compared to traditional methods, especially in large-scale genomic studies.

What Problem Does This Paper Address?

Scientific discoveries are only truly valuable if they can be replicated across different labs, populations, or time points. While meta-analysis can combine results from multiple studies, it doesn't guarantee replicability. The "partial conjunction" (PC) test offers a framework to specifically identify signals that appear in a minimum number of studies. However, when applied to high-throughput experiments (like analyzing thousands of genes), the sheer number of PC tests becomes overwhelming. Standard multiple testing corrections (like Bonferroni or Benjamini-Hochberg) become overly conservative in this scenario, leading to a severe loss of statistical power and missing genuine replicating signals. This paper aims to solve this power deficit while maintaining rigorous error control.

Methods

The core innovation is AdaFilter, an adaptive filtering multiple testing procedure. Instead of directly applying standard tests to all PC hypotheses, AdaFilter first filters the hypotheses. It identifies a subset of "least favorable nulls" – those that are most difficult to reject – and focuses the multiple testing correction on this smaller subset. This filtering is guided by a data-driven threshold, effectively borrowing information across studies to improve the estimation of the number of false discoveries.

AdaFilter proposes two main versions:

  • AdaFilter Bonferroni: Controls the Familywise Error Rate (FWER) and Per-Family Error Rate (PFER).
  • AdaFilter BH: Controls the False Discovery Rate (FDR).

The paper also introduces adjusted p-values for AdaFilter, which are computationally efficient and yield the same rejection decisions as the primary procedures.

Datasets

The paper evaluates AdaFilter using three real-world biological datasets:

  1. Duchenne Muscular Dystrophy (DMD) microarray studies: Four independent datasets were used to find differentially expressed genes that replicate across studies.
  2. Single-cell RNA sequencing (scRNA-seq) of T cells in lung cancer tumors: Data was used to identify marker genes for a specific T cell subtype that consistently show high expression across patients.
  3. Genome-Wide Association Study (GWAS) for metabolomics: Data was used to find genetic loci (SNPs) associated with metabolic traits that replicate across individuals.

Key Insights and Ideas

  • The "composite null" is the culprit: The PC null hypothesis is "composite" because it can be true in multiple ways (e.g., fewer than r studies show a signal). Standard methods conservatively assume the worst-case scenario, leading to low power.
  • Adaptive filtering as a power booster: By intelligently filtering out hypotheses that are unlikely to be true signals, AdaFilter reduces the burden of multiple testing correction, thereby increasing power.
  • Borrowing strength across studies: AdaFilter leverages the data from all studies to inform the filtering process, making it more efficient than analyzing each study in isolation.
  • "Lack of complete monotonicity" is a feature, not a bug: AdaFilter doesn't strictly adhere to the property where reducing a p-value never leads to a rejection being switched to acceptance. This seemingly undesirable trait is actually key to its power gain, allowing it to adapt to complex signal configurations.
  • Robustness to dependence: While theoretical guarantees often rely on independence across studies, simulations show AdaFilter performs well even with some dependence within studies.

Key Results

  • Increased Power: AdaFilter consistently demonstrated higher power (recall/sensitivity) compared to the "direct approach" (applying standard methods to PC p-values) and other benchmark methods like repfdr, especially in scenarios with sparse signals or large numbers of hypotheses.
  • Error Rate Control: AdaFilter procedures were shown to control FWER, PFER, and FDR at nominal levels in simulations and theoretical proofs, under specified conditions (e.g., independence across studies).
  • Real-world applications: AdaFilter successfully identified known marker genes in DMD studies and potential drug targets in metabolomics GWAS, showcasing its practical utility. For instance, in the DMD study, AdaFilter identified 32 significant genes for r=4, while a standard BH on Fisher's combined p-values only found two.

Limitations and Future Work

  • Independence across studies: The theoretical validity of AdaFilter relies on the independence of base p-values across studies. While simulations suggest robustness to within-study dependence, across-study dependence could violate Lemma 4.1.
  • Variable r and n: While AdaFilter can handle varying numbers of studies (n) and replication thresholds (r) for different genes, the current formulation requires specifying r beforehand. Data-dependent selection of r would require an additional layer of multiplicity adjustment.
  • Future work: The authors suggest exploring the application of AdaFilter to other fields beyond genomics, investigating its performance with different dependence structures, and developing methods for data-dependent selection of r.

Summary updated

Abstract

Replicability is a fundamental quality of scientific discoveries: we are interested in those signals that are detectable in different laboratories, different populations, across time etc. Unlike meta-analysis which accounts for experimental variability but does not guarantee replicability, testing a partial conjunction (PC) null aims specifically to identify the signals that are discovered in multiple studies. In many contemporary applications, for example, comparing multiple high-throughput genetic experiments, a large number M of PC nulls need to be tested simultaneously, calling for a multiple comparisons correction. However, standard multiple testing adjustments on the M PC p-values can be severely conservative, especially when M is large and the signals are sparse. We introduce AdaFilter, a new multiple testing procedure that increases power by adaptively filtering out unlikely candidates of PC nulls. We prove that AdaFilter can control FWER and FDR as long as data across studies are independent, and has much higher power than other existing methods. We illustrate the application of AdaFilter with three examples: microarray studies of Duchenne muscular dystrophy, single-cell RNA sequencing of T cells in lung cancer tumors and GWAS for metabolomics.
BibTeX
@article{wang2022detecting,
  author = {Jingshu Wang and Lin Gui and Weijie Su and Chiara Sabatti and Art B. Owen},
  title = {{Detecting multiple replicating signals using adaptive filtering procedures}},
  journal = {The Annals of Statistics},
  year = {2022},
  doi = {10.1214/21-aos2139},
}
Figure