Abstract
In high-dimensional multiple testing, researchers often face the challenge of "selective inference," where hypotheses are chosen for reporting after observing the data. This paper presents a novel framework for post hoc false positive control that leverages spatial or hierarchical structures within the data. By defining forest-structured reference families and employing local calibration via the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality, we derive confidence bounds that hold simultaneously over all possible subsets of hypotheses. Our approach significantly improves upon traditional structure-blind methods like the Simes inequality when signals are locally clustered. We provide a low-complexity recursive algorithm for practical implementation, available in the R package sansSouci.
Introduction: The Challenge of Selective Inference
Modern data analysis in fields such as genomics and neuroimaging involves testing thousands of hypotheses simultaneously. A significant risk in these high-dimensional settings is "data snooping"—the practice of selecting specific subsets of results to report based on their observed significance. Standard multiple testing procedures, such as the Benjamini-Hochberg method, typically require the selection rule to be fixed a priori. If a researcher identifies a "hot spot" in the data and decides to focus on it post hoc, the traditional statistical guarantees are no longer valid.
"Post hoc" inference addresses this by providing confidence bounds that remain valid even when the subset $S$ of interest is chosen arbitrarily after seeing the data. The goal is to state, with a pre-specified confidence level (e.g., 95%), that the number of false positives in any selected subset $S$ does not exceed a certain bound. However, existing post hoc methods often ignore the inherent physical or logical structure of the hypotheses, leading to overly conservative and loose bounds.
Methodology
The proposed method builds upon the general framework of Blanchard et al. (2017), introducing structural priors to tighten the inference.
Forest-Structured Reference Families
The core innovation is the use of a "reference family" of sets $R_k$ that follow a forest structure. In this structure, any two sets in the family are either disjoint or nested, making it ideal for representing spatial grids or hierarchical trees.
By organizing hypotheses into this hierarchy, we can perform local calibration. Instead of a global correction across all tests, we use the DKW inequality to estimate the number of true null hypotheses within each specific node of the forest. This allows the model to adapt to local signal density, providing more sensitive bounds in regions with high signal concentration.
The Algorithm
To make this approach computationally feasible for high-dimensional data, we developed a recursive, bottom-up algorithm. The complexity of this computation is $O(Hm)$, where $H$ is the depth of the tree and $m$ is the total number of hypotheses. This efficiency ensures that the method is "completely operational" for datasets with millions of variables.
The algorithm computes the optimal bound for any arbitrary subset $S$ by interpolating information from the pre-defined reference family. The term $\zeta_k$ in the algorithm represents the local calibration threshold derived from the DKW inequality.
Key Insights and Results
Through extensive synthetic simulations—including Gaussian one-sided settings with structured signals—the paper demonstrates several key advantages of the forest-based approach:
- Superiority in Clustered Signals: When signals are locally clustered (e.g., in spatial blocks), the tree-based bound provides significantly tighter lower bounds on the number of true discoveries compared to the classic Simes inequality.
- Scale Sensitivity: While the Simes bound is effective at pinpointing a few isolated, strong signals, the new DKW-tree bound excels at estimating the proportion of signal within larger, structured regions.
- Hybridization Strategy: Recognizing the complementary strengths of these methods, we propose a hybrid bound. By splitting the significance level $\alpha$, we can create an envelope that captures the precision of Simes at small scales and the power of the tree-based approach at larger scales.
- Asymptotic Gains: Theoretical analysis shows that in asymptotic settings, the proposed bound can improve upon Simes by a factor related to the signal proportion $r$ in the region, specifically $(1-r)$.
Conclusion and Future Work
This research provides a robust and computationally efficient tool for researchers to explore high-dimensional data without sacrificing statistical rigor. The implementation in the sansSouci R package facilitates its adoption in practical workflows.
Current limitations include the assumption of independent p-values for the DKW calibration. While extensions to dependent data are possible, they currently involve higher computational costs. Future work will focus on refining these dependence-robust versions and developing automated methods for choosing the optimal forest structure and hybrid parameters without manual tuning.
Summary updated