Factors Influencing the Statistical Power of Complex Data Analysis Protocols for Molecular Signature Development from Microarray Data

Constantin Aliferis, Alexander Statnikov, Ioannis Tsamardinos, Jonathan S. Schildcrout, Bryan E. Shepherd, Frank E. HarrellPLoS 2009 · DOI

Abstract

The findings of the present study have two important practical implications: First, high-throughput studies, by avoiding under-powered data analysis protocols, can achieve substantial economies in the sample size required to demonstrate the statistical significance of a predictive signal. Factors that affect power are identified and studied, showing that much less sample than previously thought may be sufficient for exploratory studies as long as these factors are considered during analysis design. Second, previous highly-cited claims that microarray assays may not be able to predict disease outcomes better than chance are shown to be due to under-powered data analysis combined with inappropriate statistical tests.

Introduction

In bioinformatics, high-throughput microarray data is frequently used to develop multivariate molecular signatures, such as predicting patient response to specific cancer treatments. However, a highly cited prior study (referred to as "Protocol I") analyzed several major datasets and concluded that microarrays could not predict cancer outcomes better than chance in 5 out of 7 cases. This conclusion cast doubt on the utility of microarray technology and suggested that massive sample sizes—potentially in the thousands—were required to find reproducible signals.

This paper addresses a fundamental question: Is this lack of predictive power an intrinsic limitation of microarray assays, or is it an artifact of inefficient, under-powered statistical and machine learning protocols?

Methodology: Comparing Analytical Protocols

The authors investigate the statistical power of complex data analysis protocols using theoretical analysis, simulated data, and a re-analysis of seven publicly available human gene expression datasets. They compare two distinct analytical pipelines:

  • Protocol I (Prior Literature): Utilizes a non-balanced holdout estimator (randomly splitting data without preserving class ratios), the proportion of misclassifications (accuracy) as an error metric, and a nearest-centroid classifier.
  • Protocol II (Proposed/Optimized): Utilizes event-balanced repeated 10-fold cross-validation, the Area Under the Receiver Operating Characteristic Curve (AUC) as the error metric, and Support Vector Machines (SVMs) as the classifier.

To rigorously test for statistical significance, Protocol II employs an outcome-value permutation test (Monte-Carlo permutation), which shuffles target labels to build a null distribution of "no predictive signal" to compare against the actual model's performance.

The Four Pillars of Statistical Power

The authors identify four specific compounding analytical flaws that severely degrade statistical power in Protocol I:

  1. Error Metric: Discontinuous metrics like accuracy are "improper scoring rules" that fail to capture prediction uncertainty. AUC is a far more powerful metric for detecting signal.
  2. Classifier Choice: SVMs offer stronger predictive ability and robustness in high-dimensional spaces compared to simpler models like nearest-centroid classifiers.
  3. Error Estimator: Holdout estimators suffer from high variance. Repeated 10-fold cross-validation drastically reduces split and internal variance, leading to higher statistical power.
  4. Event Balancing: Failing to maintain the ratio of events to non-events (class prevalence) between training and testing sets distorts the distribution and cripples classifier performance.

Results and Direct Comparison

The study re-analyzed seven datasets (including Beer et al., Veer et al., and Rosenwald et al.) to provide a direct comparison with previous findings.

  • Simulation Results: In data with a known moderate-strength signal, Protocol I had a dismal statistical power (ranging from <0.002 to 0.3). In contrast, the optimized Protocol II achieved a power of 0.93.
  • Real-World Performance: Protocol II successfully detected statistically significant predictive signals ($p < 0.05$) in 6 out of 7 real microarray datasets. The prior study, using Protocol I, only found significance in 2 out of 7.
  • Predictive Strength: The repeated 10-fold CV AUC estimates for the significant datasets ranged from 0.67 to 0.76, proving that moderate-strength biological signals can be reliably detected with modest sample sizes if the right methods are used.

Conclusion

This research demonstrates that the failure to find significant molecular signatures is often a result of the analysis protocol rather than the data itself. By optimizing error metrics, classifiers, and estimation techniques, researchers can find significant signals in existing datasets that were previously thought to contain only noise. While increasing sample size is beneficial, optimizing the analysis protocol is a highly effective and cost-efficient alternative for biomarker discovery.

Summary updated

Abstract

THE FINDINGS OF THE PRESENT STUDY HAVE TWO IMPORTANT PRACTICAL IMPLICATIONS: First, high-throughput studies by avoiding under-powered data analysis protocols, can achieve substantial economies in sample required to demonstrate statistical significance of predictive signal. Factors that affect power are identified and studied. Much less sample than previously thought may be sufficient for exploratory studies as long as these factors are taken into consideration when designing and executing the analysis. Second, previous highly-cited claims that microarray assays may not be able to predict disease outcomes better than chance are shown by our experiments to be due to under-powered data analysis combined with inappropriate statistical tests.
BibTeX
@inproceedings{aliferis2009factors,
  author = {Constantin Aliferis and Alexander Statnikov and Ioannis Tsamardinos and Jonathan S. Schildcrout and Bryan E. Shepherd and Frank E. Harrell},
  title = {{Factors Influencing the Statistical Power of Complex Data Analysis Protocols for Molecular Signature Development from Microarray Data}},
  booktitle = {PLoS},
  year = {2009},
  doi = {10.1371/journal.pone.0004922},
}
Figure