Revisiting inconsistency in large pharmacogenomic studies

Zhaleh Safikhani, Petr Smirnov, Mark Freeman, Nehmé El-Hachem, Adrian She, Rene Quevedo, Anna Goldenberg, Nicolai J. Birkbak, Christos Hatzis, Leming Shi, Andrew H. Beck, Hugo J.w.l. Aerts, John Quackenbush, Benjamin Haibe‐Kains

Abstract

In 2013, a comparative analysis of mutation and gene expression profiles and drug sensitivity measurements for 15 drugs across 471 cancer cell lines revealed substantial inconsistency in the drug responses reported by the Genomics of Drug Sensitivity in Cancer (GDSC) and Cancer Cell Line Encyclopedia (CCLE) projects. This re-analysis addresses feedback and incorporates expanded data, focusing on the distinction between targeted therapies and broad cytotoxic drugs, and the consistency of molecular profiles versus pharmacological phenotypes. Our findings confirm that while gene expression data are significantly more consistent, drug sensitivity measurements continue to pose challenges for robust biomarker discovery. The use of new statistics allowed the identification of five drugs with moderate to good consistency, but inconsistencies remain for the majority. Overall, this work emphasizes the need for experimental standardization and validation to advance the use of large pharmacogenomic screens.

Introduction: The Reproducibility Challenge in Pharmacogenomics

The promise of precision oncology relies on identifying molecular biomarkers that predict drug sensitivity. Large-scale screens like the GDSC and CCLE are foundational to this effort. However, initial comparisons suggested significant disagreements between these datasets regarding which cell lines respond to specific compounds. This study revisits those findings using updated data and refined statistical methodologies to determine if these inconsistencies are inherent to the biological assays or a result of methodological differences.

Methods and Data Integration

To ensure a rigorous comparison, the researchers developed PharmacoGx, an open-source R package designed to standardize the processing of large-scale pharmacogenomic data. The analysis focused on 15 drugs and 698 cell lines common to both the July 2015 GDSC release and the February 2015 CCLE release.

The PharmacoGx pipeline for standardized data curation and consistency analysis.

Key methodological improvements included:

  • Standardized Curve Fitting: Raw dose-response data were re-fitted using a consistent three-parameter sigmoid model to eliminate discrepancies introduced by different original mathematical models.
  • Area Between Curves (ABC): This metric was employed to quantify geometric differences between response curves over identical concentration ranges.
  • SNP Fingerprinting: Genetic identity was verified across studies to rule out cell line drift or mislabeling.

Key Insights and Results

The re-analysis highlighted a stark contrast between molecular and phenotypic robustness. While gene expression and copy number data showed high consistency (Spearman correlation > 0.8), drug sensitivity measurements remained highly sensitive to experimental noise.

Targeted vs. Broad-Effect Drugs

Consistency was notably higher for highly targeted therapies, such as PLX4720 (targeting BRAF mutations), where the "all-or-nothing" biological response provides a signal strong enough to overcome experimental variability. In contrast, broad cytotoxic drugs exhibited significant noise, likely due to differences in experimental conditions such as seeding density or growth media.

Biomarker Validation

Out of the 15 drugs studied, only five (Nilotinib, Crizotinib, PLX4720, 17-AAG, and PD-0325901) showed moderate to good consistency. However, known biomarkers were successfully validated in both datasets. By applying more stringent statistics, the authors achieved a >75% validation rate for the most significant biomarkers of eight drugs, a marked improvement over previous analyses.

Discussion and Future Directions

Despite improvements in validation rates, the study underscores that many discovered associations remain dataset-specific. Limitations include the lack of raw technical triplicate data from CCLE and the inherent arbitrariness of sensitivity thresholds (e.g., IC50 cutoffs).

The authors conclude that the path forward for precision medicine requires the community to adopt standardized experimental and computational pipelines. Until laboratory protocols for drug screening are harmonized, the potential for large-scale pharmacogenomics to drive clinical breakthroughs will remain constrained by the challenges of reproducibility.

Summary updated

Abstract

<ns4:p>In 2013, we published a comparative analysis mutation and gene expression profiles and drug sensitivity measurements for 15 drugs characterized in the 471 cancer cell lines screened in the Genomics of Drug Sensitivity in Cancer (GDSC) and Cancer Cell Line Encyclopedia (CCLE). While we found good concordance in gene expression profiles, there was substantial inconsistency in the drug responses reported by the GDSC and CCLE projects. We received extensive feedback on the comparisons that we performed. This feedback, along with the release of new data, prompted us to revisit our initial analysis. Here we present a new analysis using these expanded data in which we address the most significant suggestions for improvements on our published analysis — that targeted therapies and broad cytotoxic drugs should have been treated differently in assessing consistency, that consistency of both molecular profiles and drug sensitivity measurements should both be compared across cell lines, and that the software analysis tools we provided should have been easier to run, particularly as the GDSC and CCLE released additional data.</ns4:p> <ns4:p> </ns4:p> <ns4:p>Our re-analysis supports our previous finding that gene expression data are significantly more consistent than drug sensitivity measurements. The use of new statistics to assess data consistency allowed us to identify two broad effect drugs and three targeted drugs with moderate to good consistency in drug sensitivity data between GDSC and CCLE. For three other targeted drugs, there were not enough sensitive cell lines to assess the consistency of the pharmacological profiles. We found evidence of inconsistencies in pharmacological phenotypes for the remaining eight drugs.</ns4:p> <ns4:p> Overall, our findings suggest that the drug sensitivity data in GDSC and CCLE continue to present challenges for robust biomarker discovery. This re-analysis provides additional support for the argument that experimental standardization and validation of pharmacogenomic response will be necessary to advance the broad use of large pharmacogenomic screens.</ns4:p>
BibTeX
@inproceedings{safikhani2017revisiting,
  author = {Zhaleh Safikhani and Petr Smirnov and Mark Freeman and Nehmé El-Hachem and Adrian She and Rene Quevedo and Anna Goldenberg and Nicolai J. Birkbak and Christos Hatzis and Leming Shi and Andrew H. Beck and Hugo J.W.L. Aerts and John Quackenbush and Benjamin Haibe‐Kains},
  title = {{Revisiting inconsistency in large pharmacogenomic studies}},
  booktitle = {F1000Research},
  year = {2017},
  doi = {10.12688/f1000research.9611.3},
}
Figure