Abstract
Constructing an accurate predictive model for clinical decision-making on the basis of a relatively small number of tumor samples with high-dimensional microarray data remains a very challenging problem. The validity of such models has been seriously questioned due to their failure in clinical validation using independent samples. Besides the statistical issues such as selection bias, some studies further implied the probable reason was improper sample selection that did not resemble the genomic space defined by the training population. Assuming that predictions would be more reliable for interpolation than extrapolation, we set to investigate the impact of applicability domain (AD) on model performance in microarray-based genomic research by evaluating and comparing model performance for samples with different extrapolation degrees. We found that the issue of applicability domain may not exist in microarray-based genomic research for clinical applications. Therefore, it is not practicable to improve model validity based on applicability domain.
Introduction
In many fields of machine learning, such as Quantitative Structure-Activity Relationship (QSAR) modeling in chemistry, it is a fundamental assumption that a model is only reliable when predicting data similar to its training set. This concept, known as the "Applicability Domain" (AD), defines a "safe zone" for predictions. When a new sample falls outside this domain, the model is considered to be extrapolating, and its predictions are often discarded as unreliable.
Predictive models built on gene expression microarrays frequently perform well during internal validation but fail to maintain that performance in independent clinical trials. A leading hypothesis for this failure has been the AD problem: the idea that validation samples are too genetically distinct from the training population. This study rigorously tests whether the AD principle applies to high-dimensional genomic research and whether quantifying a patient's "distance" from the training data can predict model failure.
Methodology
To investigate the existence of an Applicability Domain, the researchers developed a statistical framework to measure domain extrapolation:
- Distance Metric: Euclidean distance was employed to calculate the distance of a validation sample from the mean of the training set across all genes. This value was normalized into a "percentage of extrapolation" ($d_i$).
- Classification Algorithms: To ensure findings were not algorithm-dependent, three distinct methods were tested: Nearest-Centroid (NC), k-Nearest Neighbor (kNN), and Support Vector Machines (SVM).
- Validation Strategy: A 500-run Monte Carlo cross-validation approach was used. In each run, data was split into training and validation sets. Classifiers were evaluated using the Matthews Correlation Coefficient (MCC) across four extrapolation categories: 0%, <10%, 10-20%, and >20%.
Data Sources
The study utilized nine major datasets, primarily sourced from the MicroArray Quality Control (MAQC) Phase II project. These included clinical datasets for Breast Cancer (BR), Multiple Myeloma (MM), Neuroblastoma (NB), Non-Hodgkin Lymphoma (NHL), and Hepatocellular Carcinoma (HCC). Endpoints ranged from treatment response (pCR) to overall and event-free survival. Additionally, "Positive Controls" (e.g., gender prediction) and "Negative Controls" (randomly generated endpoints) were used to establish performance ceilings and floors.
Results and Discussion
The Myth of the "Safe Zone"
The most significant finding was that model performance did not significantly degrade as samples moved further from the training population's genomic space. For most clinical datasets, such as MM-EFS or NB-OS, the Prediction MCC remained remarkably stable across all extrapolation categories. In some instances, performance even slightly increased for samples categorized as being "out of domain."
Algorithm Independence
The lack of an AD effect was consistent across all tested algorithms. Whether using a simple Nearest-Centroid model or a complex SVM, the trend of stable performance persisted. This suggests that the absence of a traditional Applicability Domain is an inherent property of high-dimensional genomic data rather than a byproduct of specific modeling techniques.
Interpolation vs. Extrapolation
In classical low-dimensional statistics, interpolation is inherently safer than extrapolation. However, in the context of microarrays—where thousands of genes are measured against a relatively small number of samples—the "genomic space" is so vast that the functional distinction between interpolation and extrapolation appears to vanish. The study suggests that the "curse of dimensionality" may be masking the AD effects typically seen in lower-dimensional fields like chemistry.
Conclusion
The study concludes that it is not practicable to improve the validity of microarray-based genomic models by restricting their applicability domain. Filtering out "dissimilar" patients does not reliably prevent model failure. While clinical subtypes (e.g., specific breast cancer markers) remain vital for model accuracy due to known biological boundaries, abstract "genomic space" distance does not serve as a useful proxy for prediction reliability. Future research may explore whether more biologically informed distance metrics or advanced feature reduction techniques could reveal a more nuanced AD effect.
Summary updated