Controlling False Positives in Multiple Hypothesis Testing

120 concepts121 papers1 proposal

When a scientist tests a single hypothesis at a significance level of $\alpha = 0.05$, there is a 5% chance of detecting a pattern that does not actually exist—a false positive, or Type I error. This risk is entirely manageable.

However, modern scientific datasets rarely involve just one test. Whether an algorithm is evaluating thousands of voxels in a brain scan, scanning millions of genetic variants across the human genome, or comparing dozens of machine learning models across multiple benchmark datasets, the number of simultaneous statistical comparisons can explode.

If you run 100 independent tests, each with a 5% false positive rate, the probability of making at least one false discovery jumps to over 99%:

$$1 - (1 - 0.05)^{100} \approx 0.994$$

Without intervention, pure statistical noise will inevitably be mistaken for genuine scientific discoveries. As Ming Wang notes in Addressing Common Misuses and Pitfalls of P values in Biomedical Research, this failure to account for multiple comparisons is a major driver of the modern reproducibility crisis. To combat this, researchers across disciplines rely on multiple testing corrections to preserve the integrity of their findings.


The Family-Wise Error Rate and Bonferroni's Fix

The most traditional objective in multiple testing is to control the Family-Wise Error Rate (FWER). FWER is the probability of making even a single false positive across an entire "family" of tests.

The simplest and most famous method to control FWER is the Bonferroni correction. The mathematical intuition behind it relies on the union bound from probability theory, which states that the probability of at least one event occurring is less than or equal to the sum of their individual probabilities. Therefore, if we want the total family-wise error rate to remain below $\alpha$, we can simply divide $\alpha$ by the total number of tests ($n$). The new significance threshold for each individual test becomes:

$$\alpha_{\text{new}} = \frac{\alpha}{n}$$

This conservative approach is widely used across diverse fields. For instance, Kyle Gorman et al. applied it in We Need to Talk about Standard Splits to ensure that performance gains when comparing multiple natural language processing systems were not merely artifacts of random chance. Similarly, genetic association studies, such as the work by Boyu Chen et al. in Association of fat mass and obesity-associated and retinitis pigmentosa guanosine triphosphatase (GTPase) regulator-interacting protein-1 like polymorphisms with body mass index in Chinese women and Kuanjun He et al. in CACNA1C, schizophrenia and major depressive disorder in the Han Chinese population, routinely use Bonferroni corrections to filter out false genetic links when scanning multiple variants.

The Cost of Conservatism: Holm-Bonferroni

While the Bonferroni correction guarantees strict control over false positives, it treats every test as if it is independent and equally likely to fail. This makes it highly conservative, drastically increasing the rate of false negatives (Type II errors)—meaning real discoveries are frequently discarded.

To recover statistical power without sacrificing FWER control, researchers often turn to step-down procedures like the Holm-Bonferroni method. Instead of applying a uniform penalty, the Holm-Bonferroni method sorts the $p$-values from lowest to highest. The smallest $p$-value is tested against $\alpha/n$. If it is significant, the next $p$-value is tested against a slightly relaxed threshold of $\alpha/(n-1)$, and so on.

This sequential adjustment allows researchers to maintain rigorous standards while preserving the power to detect true effects. In Multiple-Comparison Testing for Cross-Validation Path Optimization Benchmarks: A Hessian-Mismatch Diagnostic for Holm--Bonferroni Failure Modes, Jinze Yu et al. demonstrate how applying the Holm-Bonferroni method prevents optimization benchmarks from misinterpreting random noise as significant algorithmic speedups. It has also proven vital in clinical settings; Caitlin I. Stoddard et al. utilized it to control for false discoveries when tracking complex, multi-variable antibody titers over time.


Accounting for Spatial and Structural Dependencies

The math behind the Bonferroni correction assumes the worst-case scenario: that all tests are completely independent. But what happens when your tests are highly correlated?

Consider neuroimaging. In a voxel-based morphometry study, a researcher might test 100,000 tiny 3D pixels (voxels) in a brain scan to see if their volume correlates with a disease. If you apply a strict Bonferroni correction here, as Yiyun Cai notes in Hippocampal subfield morphology from first episodes of bipolar disorder type II and major depressive disorder in a drug naïve Chinese cohort, the significance threshold becomes so microscopic that almost no structural differences will survive.

Crucially, brain voxels are not independent; they are physically continuous. If a specific voxel in the prefrontal cortex shows atrophy, its immediate neighbors almost certainly will too. Treating them as independent coin flips over-penalizes the data.

Gaussian Random Fields and Cluster-Extent Thresholds

To solve this, neuroimaging researchers use Gaussian Random Field (GRF) theory. Instead of treating the brain as a collection of isolated pixels, GRF theory treats the statistical map of the brain as a continuous, smooth mathematical landscape—like a range of rolling hills.

Bonferroni (Independent Pixels)        Gaussian Random Field (Continuous Landscape)
       .  .      .  .                         
   .   .  .  .   .  .   .                          / \         / \
 . . . . . . . . . . . . . .                  ____/   \_______/   \____
[ Treats each dot as a separate test ]       [ Measures the spatial extent of peaks ]

Instead of asking "What is the probability that this specific pixel is a false positive?", GRF asks "What is the probability that a random, smooth noise field would produce a peak or cluster of this height and spatial size?"

By calculating the spatial smoothness of the data, GRF adjusts the significance threshold based on the number of resolvable independent elements (resels) rather than the raw number of voxels. As demonstrated by Meihui Qiu et al. in The Insular Subregions Topological Characteristics of Patients With Bipolar Depressive Disorder and Loufeng Zhang et al. in Electroconvulsive Therapy Regulates the Interhemispheric Functional Connectivity of the Dorsomedial Prefrontal Cortex in Depressive Patients: Evidence from 2 Independent Samples, GRF correction allows researchers to map whole-brain connectivity maps accurately without being blinded by false positives.

Furthermore, Charles Okanda Nyatega et al. show in Gray matter, white matter and cerebrospinal fluid abnormalities in Parkinson’s disease: A voxel-based morphometry study that combining FWER correction with cluster-extent thresholds—requiring a contiguous group of significant voxels to reach a certain size—ensures that regional volume differences reflect true biological pathology rather than localized noise.


False Discovery Rate: A More Flexible Alternative

In exploratory, high-dimensional data settings—such as epigenomics or broad clinical screenings—controlling the FWER can still be too restrictive. If a genomic study evaluates 500,000 methylation sites, demanding a 0% chance of a single false positive means the study will likely miss hundreds of genuine, weaker signals.

In these scenarios, researchers shift their goal from controlling the probability of any false positive to controlling the proportion of false positives among all declared discoveries. This metric is known as the False Discovery Rate (FDR).

$$\text{FDR} = \mathbb{E}\left[ \frac{\text{False Discoveries}}{\text{Total Discoveries}} \right]$$

If a researcher accepts an FDR of 0.05 and their analysis uncovers 100 significant biomarkers, they can expect that roughly 5 of those discoveries are false positives, while the remaining 95 are real. This trade-off drastically increases statistical power.

In Longitudinal associations between β-amyloid and cortical thickness in mild cognitive impairment, Elijah Mak et al. applied FDR correction to adjust $p$-values across numerous brain regions. By controlling the proportion of false discoveries rather than eliminating them entirely, the authors successfully identified subtle, non-local relationships between amyloid buildup and cortical thinning that a conservative Bonferroni correction would have completely erased.


Advanced Inference: The Multiplier Bootstrap

While parametric corrections like Bonferroni or GRF rely on strict mathematical assumptions about the distribution of data, modern computational statistics offers non-parametric alternatives. When dealing with complex, high-dimensional machine learning estimators where the underlying distribution is unknown, researchers can use simultaneous inference techniques like the multiplier bootstrap.

In DoubleML - An Object-Oriented Implementation of Double Machine Learning in Python, Philipp Bach details how the multiplier bootstrap can be used to construct joint confidence bands across multiple hypotheses simultaneously. Instead of repeatedly resampling the entire dataset—which is computationally prohibitive when paired with heavy machine learning models—the multiplier bootstrap resamples the derivatives or influence functions of the estimators using random noise multipliers. This allows researchers to control the FWER and calculate adjusted $p$-values efficiently, ensuring robust causal inference even when accounting for hundreds of confounding variables.

Ultimately, choosing the right multiple testing correction requires balancing the cost of a false positive against the risk of missing a breakthrough. Whether through the strict bounds of Bonferroni, the spatial awareness of Gaussian Random Fields, or the flexible proportions of FDR, adjusting for multiple comparisons is what separates genuine scientific discovery from statistical illusion.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multiple_testing_correction_57aab925,
  title = {multiple testing correction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multiple-testing-correction/rdr_57aab9259c6644f7bff418e243cec3f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multiple_testing_correction_57aab925_v1,
  title = {multiple testing correction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multiple-testing-correction/rdr_57aab9259c6644f7bff418e243cec3f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake