In fields like neuroimaging and genomics, researchers aren't just looking for a needle in a haystack—they are looking for thousands of needles across millions of haystacks. When you run 100,000 statistical tests at once, how do you know which "discoveries" are real and which are just noise?
A new paper by Samuel Davenport and colleagues, titled "FDP control in multivariate linear models using the bootstrap," introduces a more powerful way to answer this question using a classic statistical tool: the bootstrap.
TL;DR
The authors develop a bootstrap-based method to provide tight, "post hoc" confidence bounds on the False Discovery Proportion (FDP) in complex linear models. By using the data itself to model noise dependencies, their approach is significantly more powerful than current state-of-the-art parametric methods used in brain imaging and genetics.
What Problem Does This Paper Address?
In high-dimensional science, we face a "multiple comparison" crisis. If you test 50,000 voxels in a brain scan, a standard 5% significance level will yield 2,500 false positives by pure chance.
Most researchers use False Discovery Rate (FDR) control to manage this. However, FDR only guarantees the expected proportion of errors over many hypothetical experiments. In a single, real-world study, the actual False Discovery Proportion (FDP)—the ratio of false positives in your specific results—could be much higher than the FDR suggests.
Furthermore, scientists often want to perform post hoc inference. This means they want to look at the data, identify an interesting cluster (like a specific brain region), and then ask: "How many of the voxels in this specific cluster are actually active?" Doing this naively is "circular inference" (or cherry-picking) and invalidates standard p-values. Existing methods to fix this, like All Resolutions Inference (ARI), are often overly conservative because they make rigid assumptions about how different data points correlate.
Methods
The authors propose a non-parametric approach using the residual bootstrap.
- The Model: They work within a multivariate linear model, which is the workhorse for analyzing fMRI and genomic data.
- The Bootstrap: Instead of assuming a specific mathematical distribution for the noise (like a Gaussian field), they "resample" the residuals from the model itself. This allows the method to automatically capture the complex spatial or biological dependencies unique to that specific dataset.
- JER Control: They combine the bootstrap with the concept of Joint Error Rate (JER) control. By finding a "thresholding template" that works across all possible subsets of hypotheses simultaneously, they can provide a mathematically guaranteed lower bound on the number of true discoveries in any subset the researcher chooses to look at.
- Theoretical Contribution: The paper provides a new, simpler proof for the consistency of the multivariate bootstrap in linear models using the Lindeberg Central Limit Theorem.
Datasets
The authors validated their method on two high-stakes data types:
- Neuroimaging: 3D fMRI data from 386 subjects in the Human Connectome Project (HCP), looking at brain activity contrasts related to IQ and sex.
- Genomics: A transcriptomic dataset of 135 subjects with Chronic Obstructive Pulmonary Disease (COPD), involving over 12,000 genes.
- Simulations: Extensive synthetic data using 2D stationary Gaussian random fields to test the method under varying levels of spatial smoothing and signal sparsity.
Key Insights and Ideas
- Modeling Dependence is Key: Parametric methods (like Simes-based ARI) assume a specific type of "positive dependence." If the data doesn't perfectly fit that mold, the bounds become "loose" and lose power. The bootstrap "sees" the actual dependence in the data, leading to much tighter bounds.
- Post Hoc Freedom: Because the control is simultaneous over all possible subsets, a researcher can click on any part of a brain map or any group of genes and get a reliable estimate of true discoveries without "breaking" the statistics.
- Simplicity Wins: The authors show that a "single-step" bootstrap is often nearly as good as more complex "step-down" procedures in real-world sparse data scenarios, making the method easier to implement.
Key Results
- Higher Power: In the COPD genomic dataset, the bootstrap method concluded with 90% confidence that at least 1,354 genes were active. In contrast, the standard ARI method could only guarantee 966 genes. That’s a massive increase in scientific "yield."
- Better Calibration: In simulations, the bootstrap method's error rate stayed much closer to the target (e.g., 0.1), while parametric methods were often "too safe" (error rates near 0.05), effectively wasting statistical power.
- Robustness: The method proved effective even when the data was heavily smoothed (common in fMRI), where parametric assumptions usually fail.
Limitations and Future Work
- Sample Size: As an asymptotic method, it requires a "reasonable" number of subjects. Simulations suggest it works well for $n \ge 20$, but it might be slightly unstable for very small pilot studies ($n=10$).
- Computational Cost: Bootstrapping is more computationally intensive than simple parametric formulas. The authors note the complexity is $O(nB|H|)$, where $B$ is the number of bootstraps. However, they provide a Python package (
pyperm) to handle this efficiently. - Template Optimization: The authors used a "linear template" for their bounds. They suggest that future work could involve "learning" the optimal template shape directly from the data to gain even more power.
Summary updated