Abstract
Background: In 2012, the Genomics of Drug Sensitivity in Cancer (GDSC) and the Cancer Cell Line Encyclopedia (CCLE) released massive datasets linking gene expression to drug response. However, a 2013 comparative analysis revealed a troubling lack of concordance in drug sensitivity measurements between the two studies, despite high agreement in gene expression profiles. This re-analysis incorporates community feedback and expanded datasets to determine if improved statistical methods and larger sample sizes can resolve these inconsistencies. Methods: We utilized the PharmacoGx platform to standardize dose-response curve fitting and introduced the Area Between Curves (ABC) and Matthews Correlation Coefficient (MCC) to assess consistency. We also integrated RNA-seq data to further validate molecular profiles. Results: While we identified five drugs with moderate to good consistency (17-AAG, PD-0332901, PLX4720, nilotinib, and crizotinib), eight of the fifteen drugs studied remained significantly inconsistent across platforms. Gene expression data continues to be significantly more reproducible than pharmacological phenotypes. Conclusion: Our findings reaffirm that drug sensitivity measures in large-scale screens remain challenging to replicate, emphasizing the urgent need for experimental standardization to advance precision medicine.
Introduction
The promise of precision medicine relies on the ability to identify robust biomarkers that predict drug response. Large-scale pharmacogenomic screens, such as the Genomics of Drug Sensitivity in Cancer (GDSC) and the Cancer Cell Line Encyclopedia (CCLE), serve as the primary benchmarks for these discoveries. However, the utility of these resources depends entirely on their reproducibility.
In 2013, a comparative study sent shockwaves through the community by reporting that GDSC and CCLE often disagreed on whether a specific cancer cell line was sensitive or resistant to a given drug. This "inconsistency" suggested that a biomarker discovered in one dataset might fail to replicate in another. This paper revisits that controversy, applying more nuanced statistical tools and expanded data to see if the discordance can be resolved.
Methods
To ensure a rigorous comparison, the researchers utilized PharmacoGx, an open-source computational platform designed to standardize pharmacogenomic data. The re-analysis moved beyond simple correlation coefficients, adopting several key methodological improvements:
- Standardized Curve Fitting: Rather than using the pre-calculated values provided by the original studies, the authors re-fit all drug dose-response curves using a consistent three-parameter sigmoid model.
- Area Between Curves (ABC): A new metric was introduced to quantify the physical difference between two dose-response curves, providing a more intuitive measure of disagreement than traditional $IC_{50}$ values.
- Matthews Correlation Coefficient (MCC): To address the "class imbalance" inherent in targeted therapies—where most cell lines are resistant and only a few are sensitive—the authors used MCC to assess the consistency of binary sensitivity calls.
- Drug Categorization: Drugs were categorized based on their response profiles into "No effect," "Narrow effect" (targeted), and "Broad effect" (cytotoxic) to determine if certain classes of drugs were more prone to inconsistency.
- Genomic Verification: SNP fingerprinting was employed to confirm that cell lines with the same name across both studies were indeed genetically identical.
Results
The Reliability of Molecular Inputs
The study confirmed that gene expression measurements (the molecular "input") are remarkably consistent across labs and platforms. Whether using microarrays or RNA-seq, the molecular profiles of cell lines were highly reproducible, providing a solid foundation for biomarker discovery. Gene expression consistency was statistically significantly higher than drug sensitivity consistency across all comparisons ($p < 0.05$).
The Challenge of Pharmacological Outputs
Despite the stability of the molecular data, the pharmacological "output"—how the cells actually responded to drugs—remained problematic. The analysis focused on 15 drugs common to both studies across 698 cell lines.
- The Consistent Five: By using specialized metrics like MCC, the authors identified five drugs that showed moderate to good consistency: 17-AAG, PD-0332901, PLX4720, nilotinib, and crizotinib. For these drugs, biomarkers discovered in one study had a high likelihood of being validated in the other.
- The Inconsistent Eight: For eight of the fifteen drugs, the measurements remained significantly inconsistent, regardless of the statistical metric used. In these cases, the "sensitive" cell lines in one study were frequently labeled as "resistant" in the other.
- Insufficient Data: For three targeted drugs (PHA-665752, erlotinib, and sorafenib), there were not enough sensitive cell lines screened in both studies to make a robust assessment of consistency.
Biomarker Validation
The ultimate test of consistency is biomarker validation. When using one study to discover a biomarker and the other to validate it, the success rate varied wildly. While drugs like PD-0325901 and nilotinib achieved validation rates over 80%, most other drugs hovered around 50%—no better than a coin flip.
Discussion and Conclusion
This re-analysis confirms that while some drugs show consistent profiles when analyzed with appropriate statistics, the majority of pharmacological data in these large screens is not yet robustly reproducible. The primary issue is not the math, but the underlying experimental variability. Most drugs do not achieve a 100% kill rate, and the resulting "fractional kill" makes metrics like $IC_{50}$ highly sensitive to minor variations in laboratory protocols.
To move forward, the field must prioritize experimental standardization. Just as the genomics community established rigorous protocols for microarrays a decade ago, the pharmacogenomics community must standardize drug assay conditions. Until then, the phenotypic data in GDSC and CCLE will continue to present significant challenges for the discovery of reliable biomarkers in precision oncology.
Summary updated