Correcting for selection bias via cross-validation in the classification of microarray data

Geoffrey J. McLachlan, Jonathan Chevelu, Jianke ZhuCollections 2008 · DOI · arXiv · OpenAlex

Abstract

There is increasing interest in the use of diagnostic rules based on microarray data. These rules are formed by considering the expression levels of thousands of genes in tissue samples taken on patients of known classification with respect to a number of classes, representing, say, disease status or treatment strategy. As the final versions of these rules are usually based on a small subset of the available genes, there is a selection bias that has to be corrected for in the estimation of the associated error rates. We consider the problem using cross-validation. In particular, we present explicit formulae that are useful in explaining the layers of validation that have to be performed in order to avoid improperly cross-validated estimates.

Introduction

In the field of bioinformatics, researchers frequently encounter the "large $p$, small $n$" problem: datasets containing expression levels for thousands of genes ($p$) but only a few dozen or hundred tissue samples ($n$). To develop practical diagnostic classifiers—such as those determining disease status—it is standard practice to winnow these thousands of genes down to a small subset of "marker" genes.

However, this process introduces a significant risk of selection bias. If the gene selection process is performed on the entire dataset before cross-validation (CV), the model effectively "peeks" at the answers. This leads to deceptively low error rates that do not generalize to new data. This paper quantifies this bias and establishes a rigorous framework for multi-layered cross-validation to produce honest error estimates.

Methodology

The study focuses on a common pipeline used in genomic classification:

  1. Classifier: Support Vector Machines (SVM) with a linear kernel, chosen for their robustness in high-dimensional spaces.
  2. Feature Selection: Recursive Feature Elimination (RFE). This method starts with the full gene set, fits an SVM, ranks genes by weight, and iteratively discards the least important ones.
  3. Validation Layers:
    • Internal CV: Gene selection is performed once on the full dataset. CV is then used only to test the final model. This approach is inherently biased.
  4. External CV: The entire gene selection process is re-run from scratch within every single fold of the cross-validation loop.
  5. Nested CV (Double CV): A second layer of cross-validation is added to optimize the number of genes to include, preventing bias introduced by selecting the subset size that yielded the best results post-hoc.

These methods were evaluated using landmark breast cancer datasets, including the van ’t Veer et al. (2002) set (78 samples, 5,422 genes) and the expanded van de Vijver et al. (2002) set (295 samples, 24,481 genes).

The Magnitude of Selection Bias

The most striking insight of this research is the sheer magnitude of the "peeking penalty." When gene selection is not isolated within the validation loop, the resulting error rates are drastically underestimated.

Comparison of Internal and External CV error rates across different gene subset sizes.

As demonstrated in the table above, the discrepancy between internal and external validation is severe. For a subset of 8 genes, the internal (biased) CV suggested an error rate of 0.15. However, when proper external CV was applied—re-selecting genes for each fold—the error rate jumped to 0.29. When the authors moved to a fully unbiased Nested CV to account for the selection of the subset size itself, the error rate rose further to 0.37, which matches the error rate of using the full, unselected gene set (5,422 genes).

Discussion and Limitations

While external and nested CV provide more accurate estimates, they come with challenges. Computational cost is a primary concern; running iterative methods like RFE inside a double-loop CV is resource-intensive. Furthermore, with small sample sizes ($n < 100$), even unbiased estimates exhibit high variance, making the "true" error rate difficult to pin down without larger cohorts.

The authors also warn against "preliminary screening." Many researchers use simple t-tests to select a few dozen genes before starting analysis. If this screening is not replicated inside the CV loop, the resulting bias cannot be corrected, rendering the final error estimates unreliable.

Conclusion

This work serves as a sobering reminder for the medical AI community. While molecular classification holds great promise, many early claims of near-perfect accuracy were likely artifacts of improper validation. Rigorous, multi-layered cross-validation is not just a statistical preference; it is a necessity for developing clinical tools that actually work on new patients.

Summary updated

Abstract

<!-- *** Custom HTML *** --> There is increasing interest in the use of diagnostic rules based on microarray data. These rules are formed by considering the expression levels of thousands of genes in tissue samples taken on patients of known classification with respect to a number of classes, representing, say, disease status or treatment strategy. As the final versions of these rules are usually based on a small subset of the available genes, there is a selection bias that has to be corrected for in the estimation of the associated error rates. We consider the problem using cross-validation. In particular, we present explicit formulae that are useful in explaining the layers of validation that have to be performed in order to avoid improperly cross-validated estimates.
BibTeX
@inproceedings{mclachlan2008correcting,
  author = {Geoffrey J. McLachlan and Jonathan Chevelu and Jianke Zhu},
  title = {{Correcting for selection bias via cross-validation in the classification of microarray data}},
  booktitle = {Collections},
  year = {2008},
  doi = {10.1214/193940307000000284},
}
Figure