A comprehensive error rate for multiple testing

Djalel‐Eddine Meskaldji, Dimitri Van De Ville, Jean‐Philippe Thiran, Stephan MorgenthalerStatistical 2018 · DOI

TL;DR

This paper introduces a flexible framework for controlling error rates in multiple hypothesis testing, moving beyond traditional methods like Bonferroni and Benjamini-Hochberg (BH). By proposing a new error metric called the penalized false discovery proportion (pFDR) and a corresponding class of threshold sequences, the authors offer a way to balance the trade-off between discovering true effects and controlling false ones, especially in scenarios with large numbers of tests.

What Problem Does This Paper Address?

In scientific research, we often test many hypotheses simultaneously (e.g., thousands of genes for differential expression). The challenge is that as the number of tests increases, so does the probability of falsely rejecting a true null hypothesis (a false positive). Traditional methods like the Bonferroni correction are very conservative, drastically reducing the power to detect true effects. The Benjamini-Hochberg (BH) procedure offers a better balance by controlling the False Discovery Rate (FDR), but it can still be suboptimal in certain situations, particularly when the cost of a false discovery is high or when many true effects exist. This paper addresses the need for more nuanced error control that can adapt to different research goals and the varying costs associated with false discoveries.

Methods

The core of this paper lies in introducing a new error metric and a flexible class of threshold sequences for multiple testing.

  • Penalized False Discovery Proportion (pFDR): This metric is defined as $pFDR = FP/s_R$, where $FP$ is the number of false positives and $s_R$ is a term related to the number of rejections, effectively penalizing false discoveries based on the number of rejections. The paper then focuses on controlling the expectation of this metric, $pFDR = E(pFDP)$.
  • Threshold Sequences: The authors propose a general form for threshold sequences: $t_r = s_r \alpha / m$, where $m$ is the total number of tests, $\alpha$ is the desired error control level, and $s_r$ is a non-decreasing "shape function" that determines how the threshold changes with the rank $r$ of the ordered p-values. This framework encompasses existing methods (Bonferroni, Holm, BH) as special cases and introduces new ones, like the "truncated linear" threshold.
  • Reshaping Function: For arbitrary dependence structures among p-values, a "reshaping function" $\xi(r)$ is introduced to modify the threshold sequence, ensuring control under more general conditions.

Datasets

Not discussed in the paper. The paper focuses on theoretical development and simulation studies.

Key Insights and Ideas

  • Flexibility is Key: The paper demonstrates that a one-size-fits-all approach to error control (like Bonferroni or BH) is not always optimal. By introducing a tunable threshold shape, researchers can tailor their multiple testing strategy to the specific problem, balancing power and error control.
  • The "Truncated Linear" Threshold: This new threshold type offers a compelling compromise. It behaves like a proportional threshold (similar to BH) for small ranks, allowing for good power when true effects are sparse, but then "truncates" or flattens out, providing stricter control similar to Bonferroni for larger numbers of rejections. This is particularly useful when the cost of a false discovery is high.
  • Duality of Error Rates and Thresholds: The paper highlights a deep connection between the choice of error rate to control and the shape of the threshold sequence used. This duality provides a powerful lens for understanding and developing new multiple testing procedures.
  • Beyond Expected Values: While many methods focus on controlling the expected number of false positives or discoveries, the paper also touches upon controlling tail probabilities (e.g., median of FDR), offering more robust guarantees.

Key Results

  • The proposed framework and the "truncated linear" threshold procedure are shown to control the penalized false discovery rate (pFDR) under various dependence structures (independence, positive dependence, and general dependence).
  • Simulations (Figures 2-5) demonstrate that the truncated linear procedure can achieve higher "gain" (a utility function combining true discoveries and penalizing false ones) compared to Bonferroni and BH, especially when the "price" of a false discovery ($\lambda$) is high.
  • The paper provides theoretical results showing that the BH procedure controls a specific quantity related to the false discovery rate, and the proposed methods extend this control to more general settings and error metrics.
  • Examples using gene expression data illustrate how the proposed methods can lead to a more balanced outcome, identifying more true positives while maintaining reasonable control over false positives compared to Bonferroni.

Limitations and Future Work

  • Unknown Effect Sizes and Prices: The optimal choice of threshold shape and parameters depends on unknown factors like the true effect sizes and the price of a false discovery ($\lambda$). The paper suggests that the choice of $\tau$ for the truncated linear threshold should be related to $\sqrt{m}$, but finding the truly optimal value remains a practical challenge.
  • Computational Complexity: While the paper focuses on theoretical control, the practical implementation of some procedures, especially those involving complex reshaping functions, might incur computational costs.
  • Future Work: The authors suggest exploring other reshaping functions and further investigating the duality between error rates and threshold sequences. They also hint at the potential for developing procedures that offer a mix of different error rate controls, adapting to varying costs of false discoveries. The paper also mentions that the optimal choice of the parameter $\epsilon$ in the mixed error rate metric (MER) depends on the price $\lambda$.

Summary updated

BibTeX
@inproceedings{meskaldji2018comprehensive,
  author = {Djalel‐Eddine Meskaldji and Dimitri Van De Ville and Jean‐Philippe Thiran and Stephan Morgenthaler},
  title = {{A comprehensive error rate for multiple testing}},
  booktitle = {Statistical},
  year = {2018},
  doi = {10.1007/s00362-018-1008-y},
}
Figure