Post hoc inference via joint family-wise error rate control

Gilles Blanchard, Pierre Neuvial, Étienne Roquain

Abstract

We introduce a general methodology for post hoc inference in a large-scale multiple testing framework. The approach is called "user-agnostic" in the sense that the statistical guarantee on the number of correct rejections holds for any set of candidate items selected by the user (after having seen the data). This task is investigated by defining a suitable criterion, named the joint-family-wise-error rate (JER for short). We propose several procedures for controlling the JER, with a special focus on incorporating dependencies while adapting to the unknown quantity of signal (via a step-down approach). We show that our proposed setting incorporates as particular cases a version of the higher criticism as well as the closed testing based approach of Goeman and Solari (2011). Our theoretical statements are supported by numerical experiments.

Introduction

In modern high-throughput sciences such as genomics or neuroimaging, researchers often face the challenge of performing thousands of simultaneous hypothesis tests. Traditional methods for error control, such as the False Discovery Rate (FDR) or Family-Wise Error Rate (FWER), are designed for scenarios where the set of rejected hypotheses is determined by a pre-specified algorithm. However, in practice, researchers frequently engage in "post hoc" selection—exploring the data first and then focusing on specific clusters of interest (e.g., a group of genes or a spatial region in the brain).

This "data snooping" creates a selection effect that invalidates standard statistical guarantees. If a researcher selects the most promising results after viewing the data, the nominal p-values no longer reflect the true probability of a false positive. This paper addresses this gap by introducing a "user-agnostic" framework that remains statistically valid regardless of how the user chooses their subset of results.

Methodology: Joint Family-Wise Error Rate (JER)

The core of the proposed methodology is the Joint Family-Wise Error Rate (JER). Unlike standard FWER, which controls the probability of making at least one false discovery, JER control ensures that with high probability (e.g., $1-\alpha$), the number of false positives in any subset of hypotheses $R$ is bounded by a specific function $V(R)$.

Conceptual illustration of a user-selected set R intersecting with reference sets A and B within the hypothesis space.

To implement this, the authors utilize "threshold templates" to define reference rejection regions:

  • Linear Template: A generalization of the Simes inequality.
  • Balanced Template: A novel approach using quantiles to distribute the error budget more evenly across signal strengths, preventing the smallest p-values from exhausting the entire budget.

The framework is further refined through $\lambda$-calibration, which adjusts thresholds based on the actual dependence structure of the data (using permutations), and a step-down algorithm that iteratively estimates signal density to provide tighter bounds.

Empirical Evaluation

The methodology was evaluated using synthetic datasets designed to simulate various real-world conditions, including Gaussian equi-correlated models with varying correlation levels ($\rho$) and sparsity levels ($\pi_0$ ranging from 0.8 to 0.99).

Key findings from these simulations include:

  • Adaptivity to Dependence: Under high correlation ($\rho = 0.8$), the $\lambda$-calibration method provided significantly tighter bounds than methods assuming independence.
  • Power Gains: The Balanced Template outperformed the Linear (Simes) Template in "averaged power," demonstrating a superior ability to identify true discoveries within selected sets.
  • Signal Adaptation: The step-down refinement effectively "caught up" to the oracle bound as the signal-to-noise ratio increased, adapting to the unknown sparsity of the data.

The authors have implemented these procedures in the open-source R package, sansSouci.

Discussion and Future Work

The "user-agnostic" nature of this framework provides a rigorous mathematical safety net for exploratory data analysis. By using a reference family of sets to interpolate guarantees across the entire power set of hypotheses, researchers can optimize their selection $R$ based on the data while maintaining a valid bound on false positives.

While the current work demonstrates strong performance, several areas for future research remain. These include developing data-dependent methods for choosing the size of the reference family ($K$) and extending formal proofs for $\lambda$-calibration to cover data-dependent templates. Additionally, exploring "smooth decay" in violation probabilities could offer even greater flexibility for post hoc inference.

Summary updated

Abstract

We introduce a general methodology for post hoc inference in a large-scale multiple testing framework. The approach is called "user-agnostic" in the sense that the statistical guarantee on the number of correct rejections holds for any set of candidate items selected by the user (after having seen the data). This task is investigated by defining a suitable criterion, named the joint-family-wise-error rate (JER for short). We propose several procedures for controlling the JER, with a special focus on incorporating dependencies while adapting to the unknown quantity of signal (via a step-down approach). We show that our proposed setting incorporates as particular cases a version of the higher criticism as well as the closed testing based approach of Goeman and Solari (2011). Our theoretical statements are supported by numerical experiments.
BibTeX
@article{blanchard2020post,
  author = {Gilles Blanchard and Pierre Neuvial and Étienne Roquain},
  title = {{Post hoc inference via joint family-wise error rate control}},
  journal = {The Annals of Statistics},
  year = {2020},
  doi = {10.1214/19-aos1847},
}
Figure