TL;DR
This paper introduces a new way to calculate confidence bounds for the False Discovery Proportion (FDP) when your data points aren't independent but follow a Hidden Markov Model (HMM). By leveraging the spatial or temporal structure of the data, the authors provide "post hoc" bounds that are significantly sharper and more informative than traditional methods, allowing researchers to accurately estimate the error rate in specific, data-driven selections.
What Problem Does This Paper Address?
In modern data science—especially in genomics—we often perform thousands of statistical tests simultaneously. The classic challenge is the "multiple testing problem": the more questions you ask, the more likely you are to get a "significant" result by pure chance.
Standard techniques like the Benjamini-Hochberg procedure control the False Discovery Rate (FDR), which is the average error rate across many experiments. However, practitioners often want to know the error rate for the specific set of results they just found (the False Discovery Proportion or FDP). Furthermore, they often "cherry-pick" regions of interest after looking at the data—a process called selective inference.
Existing "agnostic" methods for FDP are safe but notoriously conservative (pessimistic) because they assume the worst-case dependency between tests. This paper addresses the gap by assuming the data follows an HMM structure (where the "truth" of a hypothesis depends on its neighbors), allowing for much tighter, more useful error estimates.
Methods
The authors develop a methodology to construct confidence bounds for the FDP under an HMM framework. Their approach moves through three levels of complexity:
- The Oracle Bound: If we knew the exact parameters of the HMM (the transition probabilities between "null" and "active" states and the distribution of the data), we could calculate the exact posterior distribution of the FDP.
- The Plug-in Bound: Since we don't know the parameters, we estimate them using an Expectation-Maximization (EM) algorithm with non-parametric kernel density estimation. We then "plug" these estimates into the oracle formula.
- Bootstrap Refinements: The plug-in approach can be "over-optimistic" because it doesn't account for the error in the parameter estimation itself. The authors propose three bootstrap-based methods (Boot1, Boot2, and Boot3) that resample the data to calibrate the bounds, ensuring they remain valid even when parameters are estimated.
The key innovation is using the forward-backward algorithm (a staple of HMMs) to efficiently compute the probability of any specific subset of hypotheses being false positives.
Datasets
The paper evaluates the methods using a mix of synthetic and real-world data:
- Simulated HMM Data: Purely synthetic data used to verify the "oracle" performance and the validity of the bootstrap bounds.
- Semi-simulated Genomic Data: Created using the
acnrandjointsegR packages. This resamples real DNA copy-number profiles to create "ground truth" benchmarks that look like real biological data. - Influenza-like Illness (ILI) Data: Weekly incidence rates from the French Sentinelles Network (1985–2008), used to detect "aberrant" weeks of flu activity.
- Ovarian Cancer Study: A dataset of 13,239 genomic loci from 117 patients, comparing DNA copy-number alterations between patients with and without endometriosis.
Key Insights and Ideas
- Structure is Power: The "aha moment" is that assuming any reasonable structure (like an HMM) is vastly superior to assuming no structure. Agnostic bounds (like the Simes bound) are often so wide they are practically useless for small, interesting subsets of data.
- The "Free Lunch" of Latent Variables: In a Bayesian or latent variable setting, you can provide bounds for any selection policy without the usual "complexity penalty" (like a Bonferroni correction) required in frequentist settings, provided your model of the underlying process is accurate.
- Accounting for the "Winner's Curse": By using bootstrap methods that re-run the selection policy on resampled data, the authors successfully mitigate the bias that comes from picking the most "significant-looking" regions.
Key Results
- Sharpness: In genomic simulations, the HMM-based bounds were much closer to the true FDP than the Simes bound. For example, where Simes might say "at most 100% of these are false," the HMM bound might say "at most 10%."
- Stability: The Boot3 method emerged as a particularly strong performer, providing a "stabilized" bound that remains valid even when the selection policy is highly data-driven.
- Real-world Utility: On the ovarian cancer data, the method was able to pinpoint specific chromosomal regions (like the "short arm" of chromosome 7) where the signal was dense, providing a narrow confidence interval for the number of true discoveries.
Limitations and Future Work
- Identifiability: The model struggles if the "null" and "alternative" distributions are too similar, or if the HMM states are nearly independent. In these "singular" cases, the bootstrap might underestimate the error.
- Selection Bias via "Insider Info": The bounds assume the researcher only uses the observed data $X$ to make selections. If a researcher uses "expert knowledge" or ancillary data not included in the HMM, the bounds may become invalid.
- Asymptotic Consistency: While the authors provide a lemma for the consistency of the plug-in bound, a full theoretical proof for the bootstrap coverage in this complex setting remains an open challenge for future research.
Summary updated