Revisiting inconsistency in large pharmacogenomic studies

Zhaleh Safikhani, Mark Freeman, Petr Smirnov, Nehmé El-Hachem, Adrian She, Rene Quevedo, Anna Goldenberg, Nicolai J. Birkbak, Christos Hatzis, Leming Shi, Andrew H. Beck, Hugo J.w.l. Aerts, John Quackenbush, Benjamin Haibe‐Kains

Abstract

Background: In 2012, the Genomics of Drug Sensitivity in Cancer (GDSC) and the Cancer Cell Line Encyclopedia (CCLE) released massive datasets linking gene expression to drug response. However, a 2013 comparative analysis revealed a troubling lack of concordance in drug sensitivity measurements between the two studies, despite high agreement in gene expression profiles. This re-analysis incorporates community feedback and expanded datasets to determine if improved statistical methods and larger sample sizes can resolve these inconsistencies. Methods: We utilized the PharmacoGx platform to standardize dose-response curve fitting and introduced the Area Between Curves (ABC) and Matthews Correlation Coefficient (MCC) to assess consistency. We also integrated RNA-seq data to further validate molecular profiles. Results: While we identified five drugs with moderate to good consistency (17-AAG, PD-0332901, PLX4720, nilotinib, and crizotinib), eight of the fifteen drugs studied remained significantly inconsistent across platforms. Gene expression data continues to be significantly more reproducible than pharmacological phenotypes. Conclusion: Our findings reaffirm that drug sensitivity measures in large-scale screens remain challenging to replicate, emphasizing the urgent need for experimental standardization to advance precision medicine.

Introduction

The promise of precision medicine relies on the ability to identify robust biomarkers that predict drug response. Large-scale pharmacogenomic screens, such as the Genomics of Drug Sensitivity in Cancer (GDSC) and the Cancer Cell Line Encyclopedia (CCLE), serve as the primary benchmarks for these discoveries. However, the utility of these resources depends entirely on their reproducibility.

In 2013, a comparative study sent shockwaves through the community by reporting that GDSC and CCLE often disagreed on whether a specific cancer cell line was sensitive or resistant to a given drug. This "inconsistency" suggested that a biomarker discovered in one dataset might fail to replicate in another. This paper revisits that controversy, applying more nuanced statistical tools and expanded data to see if the discordance can be resolved.

Methods

To ensure a rigorous comparison, the researchers utilized PharmacoGx, an open-source computational platform designed to standardize pharmacogenomic data. The re-analysis moved beyond simple correlation coefficients, adopting several key methodological improvements:

  • Standardized Curve Fitting: Rather than using the pre-calculated values provided by the original studies, the authors re-fit all drug dose-response curves using a consistent three-parameter sigmoid model.
  • Area Between Curves (ABC): A new metric was introduced to quantify the physical difference between two dose-response curves, providing a more intuitive measure of disagreement than traditional $IC_{50}$ values.
  • Matthews Correlation Coefficient (MCC): To address the "class imbalance" inherent in targeted therapies—where most cell lines are resistant and only a few are sensitive—the authors used MCC to assess the consistency of binary sensitivity calls.
  • Drug Categorization: Drugs were categorized based on their response profiles into "No effect," "Narrow effect" (targeted), and "Broad effect" (cytotoxic) to determine if certain classes of drugs were more prone to inconsistency.
  • Genomic Verification: SNP fingerprinting was employed to confirm that cell lines with the same name across both studies were indeed genetically identical.

Results

The Reliability of Molecular Inputs

The study confirmed that gene expression measurements (the molecular "input") are remarkably consistent across labs and platforms. Whether using microarrays or RNA-seq, the molecular profiles of cell lines were highly reproducible, providing a solid foundation for biomarker discovery. Gene expression consistency was statistically significantly higher than drug sensitivity consistency across all comparisons ($p < 0.05$).

The Challenge of Pharmacological Outputs

Despite the stability of the molecular data, the pharmacological "output"—how the cells actually responded to drugs—remained problematic. The analysis focused on 15 drugs common to both studies across 698 cell lines.

  1. The Consistent Five: By using specialized metrics like MCC, the authors identified five drugs that showed moderate to good consistency: 17-AAG, PD-0332901, PLX4720, nilotinib, and crizotinib. For these drugs, biomarkers discovered in one study had a high likelihood of being validated in the other.
  2. The Inconsistent Eight: For eight of the fifteen drugs, the measurements remained significantly inconsistent, regardless of the statistical metric used. In these cases, the "sensitive" cell lines in one study were frequently labeled as "resistant" in the other.
  3. Insufficient Data: For three targeted drugs (PHA-665752, erlotinib, and sorafenib), there were not enough sensitive cell lines screened in both studies to make a robust assessment of consistency.

Biomarker Validation

The ultimate test of consistency is biomarker validation. When using one study to discover a biomarker and the other to validate it, the success rate varied wildly. While drugs like PD-0325901 and nilotinib achieved validation rates over 80%, most other drugs hovered around 50%—no better than a coin flip.

Discussion and Conclusion

This re-analysis confirms that while some drugs show consistent profiles when analyzed with appropriate statistics, the majority of pharmacological data in these large screens is not yet robustly reproducible. The primary issue is not the math, but the underlying experimental variability. Most drugs do not achieve a 100% kill rate, and the resulting "fractional kill" makes metrics like $IC_{50}$ highly sensitive to minor variations in laboratory protocols.

To move forward, the field must prioritize experimental standardization. Just as the genomics community established rigorous protocols for microarrays a decade ago, the pharmacogenomics community must standardize drug assay conditions. Until then, the phenotypic data in GDSC and CCLE will continue to present significant challenges for the discovery of reliable biomarkers in precision oncology.

Summary updated

Abstract

ABSTRACT Background: In 2012, two large pharmacogenomic studies, the Genomics of Drug Sensitivity in Cancer (GDSC) and Cancer Cell Line Encyclopedia (CCLE), were published, each reported gene expression data and measures of drug response for a large number of drugs and hundreds of cell lines. In 2013, we published a comparative analysis that reported gene expression profiles for the 471 cell lines profiled in both studies and dose response measurements for the 15 drugs characterized in the common cell lines by both studies. While we found good concordance in gene expression profiles, there was substantial inconsistency in the drug responses reported by the GDSC and CCLE projects. Our paper was widely discussed and we received extensive feedback on the comparisons that we performed. This feedback, along with the release of new data, prompted us to revisit our initial analysis. Here we present a new analysis using these expanded data in which we address the most significant suggestions for improvements on our published analysis: that drugs with different response characteristics should have been treated differently, that targeted therapies and broad cytotoxic drugs should have been treated differently in assessing consistency, that consistency of both molecular profiles and drug sensitivity measurements should both be compared across cell lines to accurately assess differences in the studies, that we missed some biomarkers that are consistent between studies, and that the software analysis tools we provided with our analysis should have been easier to run, particularly as the GDSC and CCLE released additional data. Methods: For each drug, we used published sensitivity data from the GDSC and CCLE to separately estimate drug dose-response curves. We then used two statistics, the area between drug dose-response curves (ABC) and the Matthews correlation coefficient (MCC), to robustly estimate the consistency of continuous and discrete drug sensitivity measures, respectively. We also used recently released RNA-seq data together with previously published gene expression microarray data to assess inter-platform reproducibility of cell line gene expression profiles. Results: This re-analysis supports our previous finding that gene expression data are significantly more consistent than drug sensitivity measurements. The use of new statistics to assess data consistency allowed us to identify two broad effect drugs — 17-AAG and PD-0332901 — and three targeted drugs — PLX4720, nilotinib and crizotinib — with moderate to good consistency in drug sensitivity data between GDSC and CCLE. Not enough sensitive cell lines were screened in both studies to robustly assess consistency for three other targeted drugs, PHA-665752, erlotinib, and sorafenib. Concurring with our published results, we found evidence of inconsistencies in pharmacological phenotypes for the remaining eight drugs. Further, to discover “consistency” between studies required the use of multiple statistics and the selection of specific measures on a case-by-case basis. Conclusion: Our results reaffirm our initial findings of an inconsistency in drug sensitivity measures for eight of fifteen drugs screened both in GDSC and CCLE, irrespective of which statistical metric was used to assess correlation. Taken together, our findings suggest that the phenotypic data on drug response in the GDSC and CCLE continue to present challenges for robust biomarker discovery. This re-analysis provides additional support for the argument that experimental standardization and validation of pharmacogenomic response will be necessary to advance the broad use of large pharmacogenomic screens.
BibTeX
@inproceedings{safikhani2017revisiting,
  author = {Zhaleh Safikhani and Mark Freeman and Petr Smirnov and Nehmé El-Hachem and Adrian She and Rene Quevedo and Anna Goldenberg and Nicolai J. Birkbak and Christos Hatzis and Leming Shi and Andrew H. Beck and Hugo J.W.L. Aerts and John Quackenbush and Benjamin Haibe‐Kains},
  title = {{Revisiting inconsistency in large pharmacogenomic studies}},
  booktitle = {F1000Research},
  year = {2017},
  doi = {10.12688/f1000research.9611.3},
}
Figure