Re-sampling strategy to improve the estimation of number of null hypotheses in FDR control under strong correlation structures

Xin Lu, David L. PerkinsBMC Bioinformatics 2007 · DOI

TL;DR

This paper tackles the challenge of accurately estimating the proportion of true null hypotheses ($\pi_0$) in large-scale genomic studies (like microarrays) where gene expression data is heavily correlated. They introduce two novel re-sampling strategies, SampS and SampG, that break these correlations to provide more stable $\pi_0$ estimates, leading to better False Discovery Rate (FDR) control and improved gene selection compared to existing methods.

What Problem Does This Paper Address?

In high-throughput biological experiments, researchers often test thousands of hypotheses simultaneously (e.g., identifying differentially expressed genes across different conditions). A key goal is to control the False Discovery Rate (FDR), which is the expected proportion of rejected null hypotheses that are actually true. To do this effectively, it's crucial to estimate $\pi_0$, the proportion of true null hypotheses among all tested hypotheses. Many existing methods for estimating $\pi_0$ and controlling FDR rely on the assumption that test statistics (like p-values) are independent or only weakly correlated. However, real-world biological data, especially from microarrays, often exhibits strong correlations between genes due to co-regulation or shared biological pathways. This strong correlation violates the independence assumption, leading to unstable and biased $\pi_0$ estimates, which in turn compromises FDR control and the ability to reliably identify true discoveries.

Methods

The authors propose two re-sampling strategies, SampS and SampG, designed to mitigate the impact of gene-gene correlations on $\pi_0$ estimation. These methods are integrated into the existing q-value framework, replacing the direct estimation of $\pi_0$ from p-values.

  • SampS (Sample Re-sampling): This method employs a model averaging approach. It repeatedly samples a subset (2/3) of the samples from each experimental group without replacement. For each resampled dataset, p-values are calculated, and then $\pi_0$ is estimated. The final $\pi_0$ estimate used is the upper quartile of these repeated estimations. This process helps to smooth out variations caused by correlations.
  • SampG (Gene Expression Value Re-sampling): This strategy goes a step further by breaking down correlations more aggressively. It repeatedly samples a subset (2/3) of the expression values for each gene independently within each class. Similar to SampS, p-values are calculated, $\pi_0$ is estimated, and the upper quartile of these estimations is used. By re-sampling individual expression values, this method aims to break the direct gene-gene correlations more effectively.

Both methods use a "plug-in" estimator for $\pi_0$ (e.g., the q-value method or the moment generating function method) on the resampled data. The choice of the upper quartile is a conservative strategy to ensure strong FDR control, even if it introduces a slight bias.

Datasets

The paper uses two types of simulated datasets and one real-world dataset for evaluation:

  • Data-B: A simulated dataset designed to mimic block-wise correlation structures. Genes are grouped into blocks, with high correlation within blocks and independence between blocks. This allows for controlled manipulation of correlation strength and structure.
  • Data-M: A simulated dataset derived from an actual human breast cancer microarray dataset (65 ER+ and 46 ER- cases). This dataset preserves the complex correlation structures found in real biological data while allowing the researchers to control the true proportion of null hypotheses ($\pi_0$).
  • Actual Microarray Dataset: A published dataset of human breast cancer samples was used to initially assess the variability of $\pi_0$ estimation methods under correlation.

Key Insights and Ideas

  • Correlation is a major disruptor: Strong gene-gene correlations significantly distort the distribution of p-values for true null hypotheses, making standard $\pi_0$ estimation methods unstable and unreliable.
  • Breaking correlations is key: Re-sampling strategies that disrupt these correlations (either by re-sampling samples or individual expression values) are crucial for robust $\pi_0$ estimation.
  • Upper quartile for conservatism: Using the upper quartile of re-sampled $\pi_0$ estimates, while potentially introducing a slight bias, provides a more conservative and reliable estimate for strong FDR control.
  • SampG offers stronger correlation breaking: The SampG method, by re-sampling individual gene expression values, is more effective at breaking down correlations than SampS, leading to lower Mean Squared Errors (MSE) in $\pi_0$ estimation.

Key Results

  • Reduced $\pi_0$ estimation variance: The proposed SampS and SampG methods significantly reduced the variation in $\pi_0$ estimates compared to standard methods like q-value, especially in the presence of strong correlations.
  • Improved FDR control: When controlling for FDR at a 0.05 level, the SampS and SampG methods resulted in a lower actual FDR compared to the q-value method.
  • Minor power trade-off: While achieving better FDR control, the proposed methods showed a minor reduction in power (ability to detect true positives) compared to the q-value method, but this reduction was significantly less than that observed with highly conservative methods like BY.
  • MSE reduction: Table 1 shows that SampG.m (SampG with mgf plug-in) generally achieved the lowest Mean Squared Error (MSE) for $\pi_0$ estimation across various simulated scenarios, outperforming other methods.
  • Significant FDR reduction: Table 4 indicates that the SampS and SampG methods could reduce the average FDR by up to 40% compared to the q-value method, with a decrease in average power of less than 1%.

Limitations and Future Work

  • Bias vs. Variance Trade-off: The authors acknowledge that using the upper quartile for $\pi_0$ estimation introduces a slight bias, although this is a deliberate choice to ensure strong FDR control.
  • Applicability to Complex Designs: While SampS (sample re-sampling) might be applicable to more complex experimental designs (like time-course studies) if stationary patterns exist, SampG (individual value re-sampling) might be less directly applicable without sufficient replicates per time point.
  • Generalizability: The authors suggest that the core idea of using re-sampling to estimate $\pi_0$ and then applying a conservative strategy (like the upper quartile) could be applied to other FDR algorithms in scenarios with strong correlations.
  • Software Availability: The R code for the proposed methods and simulated datasets is provided as supplementary material, facilitating further research and application.

Summary updated

BibTeX
@article{lu2007re,
  author = {Xin Lu and David L. Perkins},
  title = {{Re-sampling strategy to improve the estimation of number of null hypotheses in FDR control under strong correlation structures}},
  journal = {BMC Bioinformatics},
  year = {2007},
  doi = {10.1186/1471-2105-8-157},
}
Figure