Post hoc false discovery proportion inference under a Hidden Markov Model

Marie Perrot-Dockès, Gilles Blanchard, Pierre Neuvial, Étienne Roquain

Abstract

We address the multiple testing problem under the assumption that the true/false hypotheses are driven by a Hidden Markov Model (HMM), which is recognized as a fundamental setting to model multiple testing under dependence. While previous work has concentrated on deriving specific procedures with a controlled False Discovery Rate (FDR), we consider the problem of establishing confidence bounds on the false discovery proportion (FDP) for a user-selected set of hypotheses that can depend on the observed data in an arbitrary way. We develop a methodology to construct such confidence bounds first when the HMM model is known, then when its parameters are unknown and estimated using a nonparametric approach. In the latter case, we propose a bootstrap-based methodology to take into account the effect of parameter estimation error. We show that taking advantage of the assumed HMM structure allows for a substantial improvement of confidence bound sharpness over existing agnostic methods, as witnessed both via numerical experiments and real data examples.

Introduction

In modern high-throughput sciences like genomics, researchers often test thousands of hypotheses simultaneously. A common goal is to identify regions of interest, such as DNA copy-number alterations, where signals tend to be spatially clustered. Traditional multiple testing corrections often focus on the False Discovery Rate (FDR), which provides an average error rate across all tests. However, in practice, scientists often perform "selective inference"—choosing specific sets of results to follow up on after observing the data. Standard "agnostic" methods for bounding the False Discovery Proportion (FDP) in these selected sets often ignore the underlying dependency structure of the data, leading to overly conservative and less informative bounds.

Methodology

This research leverages the Hidden Markov Model (HMM) to capture spatial or temporal dependencies between hypotheses. The approach progresses through three stages of complexity:

  1. The Oracle Bound: Assuming known HMM parameters, the posterior distribution of the hidden states is used to calculate exact confidence bounds for any user-defined set.
  2. The Plug-in Bound: In real-world scenarios, parameters are unknown. The authors employ an Expectation-Maximization (EM) algorithm with non-parametric kernel density estimation to estimate the null and alternative distributions directly from the data.
  3. Bootstrap Calibration: To account for the uncertainty introduced by parameter estimation, three bootstrap strategies (Boot1, Boot2, and Boot3) are introduced. These methods resample the data to ensure the confidence bounds remain valid (maintaining frequentist coverage) even when the model is estimated.
Figure 1: Visualization of a user-selected set of hypotheses (orange) across genomic positions. The HMM framework enables researchers to calculate tight FDP bounds for such data-driven selections by accounting for spatial dependencies.

Key Insights

The primary innovation lies in using the HMM structure as a "de-noising" mechanism. By recognizing that biological signals often occur in clusters, the HMM-based bounds can distinguish between isolated noise and genuine regional signals more effectively than structure-free methods like the Simes bound.

A significant theoretical finding is that the "Oracle" version of this bound does not require a complexity penalty for the size or nature of the selected set, provided the selection depends only on the observed data. This allows for a "post hoc" flexibility that is highly valuable for exploratory data analysis.

Results and Discussion

The methodology was evaluated using synthetic simulations, semi-simulated genomic data, and real-world datasets including influenza incidence and ovarian cancer genomic profiles.

  • Sharpness and Power: In scenarios with low signal-to-noise ratios, the HMM-based bounds were significantly tighter than the agnostic Simes bound. For instance, where Simes might provide a trivial bound (near 100% FDP), the HMM approach could provide a much lower, actionable bound. The bootstrap refinements (particularly Boot2 and Boot3) maintained high power, closely matching the performance of the Oracle bound.
  • Genomic Application: On an ovarian cancer dataset involving over 13,000 genomic locations, the method successfully identified specific chromosomal regions with low FDP, providing statistical validation for data-driven discoveries that previous methods could not support with equal precision.

Conclusion

By incorporating HMM dependencies into the selective inference framework, this work provides a powerful tool for researchers to obtain sharp, valid confidence bounds on the error rates of their discoveries. While the method relies on a reasonable model fit and identifiable distributions, it represents a substantial step forward in bridging Bayesian-style posterior modeling with frequentist error control for complex, dependent data.

Summary updated

Abstract

We address the multiple testing problem under the assumption that the true/false hypotheses are driven by a Hidden Markov Model (HMM), which is recognized as a fundamental setting to model multiple testing under dependence since the seminal work of \citet{sun2009large}. While previous work has concentrated on deriving specific procedures with a controlled False Discovery Rate (FDR) under this model, following a recent trend in selective inference, we consider the problem of establishing confidence bounds on the false discovery proportion (FDP), for a user-selected set of hypotheses that can depend on the observed data in an arbitrary way. We develop a methodology to construct such confidence bounds first when the HMM model is known, then when its parameters are unknown and estimated, including the data distribution under the null and the alternative, using a nonparametric approach. In the latter case, we propose a bootstrap-based methodology to take into account the effect of parameter estimation error. We show that taking advantage of the assumed HMM structure allows for a substantial improvement of confidence bound sharpness over existing agnostic (structure-free) methods, as witnessed both via numerical experiments and real data examples.
BibTeX
@inproceedings{perrotdockes2023post,
  author = {Marie Perrot-Dockès and Gilles Blanchard and Pierre Neuvial and Étienne Roquain},
  title = {{Post hoc false discovery proportion inference under a Hidden Markov Model}},
  booktitle = {Test},
  year = {2023},
  doi = {10.1007/s11749-023-00886-7},
}
Figure