FDP control in multivariate linear models using the bootstrap

Samuel Davenport, Bertrand Thirion, Pierre Neuvial

Abstract

In high-dimensional data analysis, such as neuroimaging and genomics, researchers often require post hoc inference capabilities to assess the False Discovery Proportion (FDP) within specific subsets of hypotheses selected after data inspection. This paper introduces a novel method for FDP control in multivariate linear models using the residual bootstrap to simulate the distribution of null contrasts. By integrating the bootstrap with the post hoc inference bounds established by Blanchard et al. (2020), we provide simultaneous asymptotic control of the FDP across all possible subsets. We further establish the consistency of the multivariate bootstrap in this context using the Lindeberg Central Limit Theorem, offering a more streamlined proof than previous literature. Empirical evaluations on fMRI and transcriptomic datasets demonstrate that our bootstrap-based approach offers superior statistical power and better calibration compared to state-of-the-art parametric methods.

Introduction

In modern scientific fields like neuroimaging and genomics, the "multiple comparisons problem" is a significant hurdle. When conducting tens of thousands of statistical tests simultaneously—such as analyzing voxels in a brain scan or gene expressions in a tissue sample—the likelihood of encountering false positives is nearly certain.

Traditionally, researchers have relied on False Discovery Rate (FDR) control to manage these errors. However, FDR only guarantees the expected proportion of false discoveries over many hypothetical repetitions of an experiment. In any single study, the actual False Discovery Proportion (FDP)—the realized ratio of false positives—can vary significantly from the FDR target. Furthermore, scientists often wish to perform "post hoc" inference: identifying a cluster of interest after seeing the results and then quantifying the reliability of that specific cluster. Standard p-values are invalid in this "circular" context. Existing solutions, such as All Resolutions Inference (ARI), often rely on rigid parametric assumptions that can lead to overly conservative results when data dependencies are complex.

Methodology: The Residual Bootstrap

To address these limitations, we propose a non-parametric framework based on the residual bootstrap within a multivariate linear model. The core innovation lies in using the data itself to model noise dependencies rather than assuming a specific mathematical distribution (e.g., a Gaussian random field).

  1. The Model: We operate within the standard multivariate linear model, $Y = XB + E$, where $Y$ represents the observed data, $X$ the design matrix, and $E$ the error terms.
  2. Bootstrap Resampling: Instead of assuming a fixed covariance structure for $E$, we resample the residuals from the model. This allows the method to capture the unique spatial or biological correlations present in the specific dataset being analyzed.
  3. Joint Error Rate (JER) Control: We combine the bootstrap with Joint Error Rate control. By identifying a "thresholding template" that is valid across all possible subsets of hypotheses simultaneously, we can provide a mathematically guaranteed lower bound on the number of true discoveries in any subset chosen by the researcher.
  4. Theoretical Consistency: A key contribution of this work is a simplified proof for the consistency of the multivariate bootstrap in linear models. By utilizing the Lindeberg Central Limit Theorem, we demonstrate that the bootstrap distribution converges to the true distribution of the null contrasts as the sample size grows.

Empirical Results and Applications

We validated the bootstrap method against existing parametric approaches (such as Simes-based ARI) using both synthetic simulations and real-world high-dimensional datasets.

Genomic Analysis (COPD Dataset)

In a transcriptomic study of 135 subjects with Chronic Obstructive Pulmonary Disease (COPD) involving over 12,000 genes, our method demonstrated significantly higher discovery power. With 90% confidence, the bootstrap method identified at least 1,354 active genes, whereas the standard parametric ARI method could only guarantee 966 genes. This 40% increase in guaranteed discoveries illustrates the benefit of modeling actual data dependencies.

Neuroimaging (Human Connectome Project)

Using fMRI data from 386 subjects in the Human Connectome Project (HCP), we analyzed brain activity contrasts related to IQ and sex. The bootstrap method proved robust even under heavy spatial smoothing—a common preprocessing step in fMRI that often violates the assumptions of parametric models. Our approach maintained tight FDP bounds where other methods became excessively conservative.

Simulation Insights

Extensive simulations using 2D stationary Gaussian random fields showed that the bootstrap method's error rate remains close to the target nominal level (e.g., $\alpha = 0.1$). In contrast, parametric methods were often "too safe," with actual error rates falling near 0.05, which effectively wastes statistical power and results in fewer scientific discoveries.

Discussion and Limitations

While the bootstrap approach provides a powerful and flexible alternative to parametric FDP control, it is subject to certain constraints:

  • Sample Size: As an asymptotic method, it requires a sufficient number of observations. Our simulations suggest the method is reliable for $n \ge 20$, but may exhibit instability in very small pilot studies ($n < 10$).
  • Computational Demand: Bootstrapping is more intensive than applying parametric formulas, with a complexity of $O(nB|H|)$ (where $B$ is the number of bootstrap iterations). To mitigate this, we provide an efficient implementation via the pyperm Python package.
  • Template Optimization: Current results utilize a linear thresholding template. Future research may explore "learning" the optimal template shape directly from the data to further enhance statistical power.

Conclusion

By leveraging the residual bootstrap, we provide a robust framework for post hoc FDP control that adapts to the underlying structure of high-dimensional data. This method allows researchers the freedom to explore their data and select regions of interest without compromising statistical integrity, ultimately leading to more powerful and reliable scientific insights in fields like genomics and brain imaging.

Summary updated

Abstract

In this article we develop a method for performing post hoc inference of the False Discovery Proportion (FDP) over multiple contrasts of interest in the multivariate linear model. To do so we use the bootstrap to simulate from the distribution of the null contrasts. We combine the bootstrap with the post hoc inference bounds of Blanchard (2020) and prove that doing so provides simultaneous asymptotic control of the FDP over all subsets of hypotheses. This requires us to demonstrate consistency of the multivariate bootstrap in the linear model, which we do via the Lindeberg Central Limit Theorem, providing a simpler proof of this result than that of Eck (2018). We demonstrate, via simulations, that our approach provides simultaneous control of the FDP over all subsets and is typically more powerful than existing, state of the art, parametric methods. We illustrate our approach on functional Magnetic Resonance Imaging data from the Human Connectome project and on a transcriptomic dataset of chronic obstructive pulmonary disease.
BibTeX
@inproceedings{davenport2025fdp,
  author = {Samuel Davenport and Bertrand Thirion and Pierre Neuvial},
  title = {{FDP control in multivariate linear models using the bootstrap}},
  booktitle = {Electronic},
  year = {2025},
  doi = {10.1214/25-ejs2354},
}
Figure