Selective inference for false discovery proportion in a hidden Markov model

Marie Perrot-Dockès, Gilles Blanchard, Pierre Neuvial, Étienne Roquain

TL;DR

This paper introduces a new way to calculate confidence bounds for the False Discovery Proportion (FDP) when your data points aren't independent but follow a Hidden Markov Model (HMM). By leveraging the spatial or temporal structure of the data, the authors provide "post hoc" bounds that are significantly sharper and more informative than traditional methods, allowing researchers to accurately estimate the error rate in specific, data-driven selections.

What Problem Does This Paper Address?

In modern data science—especially in genomics—we often perform thousands of statistical tests simultaneously. The classic challenge is the "multiple testing problem": the more questions you ask, the more likely you are to get a "significant" result by pure chance.

Standard techniques like the Benjamini-Hochberg procedure control the False Discovery Rate (FDR), which is the average error rate across many experiments. However, practitioners often want to know the error rate for the specific set of results they just found (the False Discovery Proportion or FDP). Furthermore, they often "cherry-pick" regions of interest after looking at the data—a process called selective inference.

Existing "agnostic" methods for FDP are safe but notoriously conservative (pessimistic) because they assume the worst-case dependency between tests. This paper addresses the gap by assuming the data follows an HMM structure (where the "truth" of a hypothesis depends on its neighbors), allowing for much tighter, more useful error estimates.

Methods

The authors develop a methodology to construct confidence bounds for the FDP under an HMM framework. Their approach moves through three levels of complexity:

  1. The Oracle Bound: If we knew the exact parameters of the HMM (the transition probabilities between "null" and "active" states and the distribution of the data), we could calculate the exact posterior distribution of the FDP.
  2. The Plug-in Bound: Since we don't know the parameters, we estimate them using an Expectation-Maximization (EM) algorithm with non-parametric kernel density estimation. We then "plug" these estimates into the oracle formula.
  3. Bootstrap Refinements: The plug-in approach can be "over-optimistic" because it doesn't account for the error in the parameter estimation itself. The authors propose three bootstrap-based methods (Boot1, Boot2, and Boot3) that resample the data to calibrate the bounds, ensuring they remain valid even when parameters are estimated.

The key innovation is using the forward-backward algorithm (a staple of HMMs) to efficiently compute the probability of any specific subset of hypotheses being false positives.

Datasets

The paper evaluates the methods using a mix of synthetic and real-world data:

  • Simulated HMM Data: Purely synthetic data used to verify the "oracle" performance and the validity of the bootstrap bounds.
  • Semi-simulated Genomic Data: Created using the acnr and jointseg R packages. This resamples real DNA copy-number profiles to create "ground truth" benchmarks that look like real biological data.
  • Influenza-like Illness (ILI) Data: Weekly incidence rates from the French Sentinelles Network (1985–2008), used to detect "aberrant" weeks of flu activity.
  • Ovarian Cancer Study: A dataset of 13,239 genomic loci from 117 patients, comparing DNA copy-number alterations between patients with and without endometriosis.

Key Insights and Ideas

  • Structure is Power: The "aha moment" is that assuming any reasonable structure (like an HMM) is vastly superior to assuming no structure. Agnostic bounds (like the Simes bound) are often so wide they are practically useless for small, interesting subsets of data.
  • The "Free Lunch" of Latent Variables: In a Bayesian or latent variable setting, you can provide bounds for any selection policy without the usual "complexity penalty" (like a Bonferroni correction) required in frequentist settings, provided your model of the underlying process is accurate.
  • Accounting for the "Winner's Curse": By using bootstrap methods that re-run the selection policy on resampled data, the authors successfully mitigate the bias that comes from picking the most "significant-looking" regions.

Key Results

  • Sharpness: In genomic simulations, the HMM-based bounds were much closer to the true FDP than the Simes bound. For example, where Simes might say "at most 100% of these are false," the HMM bound might say "at most 10%."
  • Stability: The Boot3 method emerged as a particularly strong performer, providing a "stabilized" bound that remains valid even when the selection policy is highly data-driven.
  • Real-world Utility: On the ovarian cancer data, the method was able to pinpoint specific chromosomal regions (like the "short arm" of chromosome 7) where the signal was dense, providing a narrow confidence interval for the number of true discoveries.

Limitations and Future Work

  • Identifiability: The model struggles if the "null" and "alternative" distributions are too similar, or if the HMM states are nearly independent. In these "singular" cases, the bootstrap might underestimate the error.
  • Selection Bias via "Insider Info": The bounds assume the researcher only uses the observed data $X$ to make selections. If a researcher uses "expert knowledge" or ancillary data not included in the HMM, the bounds may become invalid.
  • Asymptotic Consistency: While the authors provide a lemma for the consistency of the plug-in bound, a full theoretical proof for the bootstrap coverage in this complex setting remains an open challenge for future research.

Summary updated

Abstract

We address the multiple testing problem under the assumption that the true/false hypotheses are driven by a Hidden Markov Model (HMM), which is recognized as a fundamental setting to model multiple testing under dependence since the seminal work of \citet{sun2009large}. While previous work has concentrated on deriving specific procedures with a controlled False Discovery Rate (FDR) under this model, following a recent trend in selective inference, we consider the problem of establishing confidence bounds on the false discovery proportion (FDP), for a user-selected set of hypotheses that can depend on the observed data in an arbitrary way. We develop a methodology to construct such confidence bounds first when the HMM model is known, then when its parameters are unknown and estimated, including the data distribution under the null and the alternative, using a nonparametric approach. In the latter case, we propose a bootstrap-based methodology to take into account the effect of parameter estimation error. We show that taking advantage of the assumed HMM structure allows for a substantial improvement of confidence bound sharpness over existing agnostic (structure-free) methods, as witnessed both via numerical experiments and real data examples.
BibTeX
@inproceedings{perrotdockes2023selective,
  author = {Marie Perrot-Dockès and Gilles Blanchard and Pierre Neuvial and Étienne Roquain},
  title = {{Selective inference for false discovery proportion in a hidden Markov model}},
  booktitle = {Test},
  year = {2023},
  doi = {10.1007/s11749-023-00886-7},
}
Figure