A comprehensive assessment of somatic mutation detection in cancer using whole-genome sequencing

Tyler Alioto, Ivo Buchhalter, Sophia Derdak, Barbara Hutter, Matthew Eldridge, Eivind Hovig, Lawrence E. Heisler, Timothy A. Beck, Jared T. Simpson, Laurie Tonon, Anne-Sophie Sertier, Ann‐Marie Patch, Natalie Jäger, Philip Ginsbach, Ruben M. Drews, Nagarajan Paramasivam, Rolf Kabbe, Sasithorn Chotewutmontri, Nicolle Diessl, Christopher Previti, Sabine Schmidt, Benedikt Brors, Lars Feuerbach, Michael C. Heinold, Susanne Gröbner, Andrey Korshunov, Patrick Tarpey, Adam P. Butler, Jonathan Hinton, David Jones, Andrew Menzies, Keiran Raine, Rebecca Shepherd, Lucy Stebbings, Jon W. Teague, Paolo Ribeca, Francesc Castro-Giner, Sergi Beltrán, Emanuele Raineri, Marc Dabad, Simon Heath, Marta Gut, Robert E. Denroche, Nicholas J. Harding, Takafumi N. Yamaguchi, Akihiro Fujimoto, Hidewaki Nakagawa, Vı́ctor Quesada, Rafael Valdés‐Mas, Sigve Nakken, Daniel Vodák, Lawrence Bower, Andy G. Lynch, Charlotte Anderson, Nicola Waddell, John V. Pearson, Sean M. Grimmond, Myron Peto, Paul T. Spellman, Minghui He, Cyriac Kandoth, Semin Lee, John H. Zhang, Louis Létourneau, Singer Ma, Sahil Seth, David Torrents, Xi Liu, David A. Wheeler, Carlos López-Otı́n, Elı́as Campo, Peter J. Campbell, Paul C. Boutros, Xosé S. Puente, Daniela S. Gerhard, Stefan M. Pfister, John D. McPherson, Thomas J. Hudson, Matthias Schlesner, Peter Lichter, Roland EilsNature Communications 2015 · DOI

Abstract

As whole-genome sequencing for cancer genome analysis becomes a clinical tool, a full understanding of the variables affecting sequencing analysis output is required. Here using tumour-normal sample pairs from two different types of cancer, chronic lymphocytic leukaemia and medulloblastoma, we conduct a benchmarking exercise within the context of the International Cancer Genome Consortium. We compare sequencing methods, analysis pipelines and validation methods. We show that using PCR-free methods and increasing sequencing depth to ∼ 100 × shows benefits, as long as the tumour:control coverage ratio remains balanced. We observe widely varying mutation call rates and low concordance among analysis pipelines, reflecting the artefact-prone nature of the raw data and lack of standards for dealing with the artefacts. However, we show that, using the benchmark mutation set we have created, many issues are in fact easy to remedy and have an immediate positive impact on mutation detection accuracy.

Introduction

Whole-genome sequencing (WGS) is rapidly transitioning from a research tool to a clinical necessity in oncology. However, the lack of standardized protocols for library preparation, sequencing depth, and bioinformatic analysis has historically created a "wild west" environment for genomic data. This study, conducted by the International Cancer Genome Consortium (ICGC), provides a critical benchmarking exercise to identify and understand the key variables that dictate the accuracy of somatic mutation detection.

Methods and Datasets

The researchers utilized tumor-normal pairs from two distinct cancer types: chronic lymphocytic leukaemia (CLL) and medulloblastoma (MB). To establish a "ground truth," samples were sequenced at an ultra-high depth of approximately 300x across five different centers. This massive dataset was manually curated by experts to create a "Gold Set" of somatic single-base mutations (SSMs) and small insertions/deletions (SIMs). This benchmark was then used to evaluate 18 different SSM and 16 different SIM detection pipelines, testing variables such as sequencing depth (ranging from 20x to 250x) and tumor purity.

Results

The PCR-Free Advantage and Coverage Evenness

A primary finding was the significant impact of library preparation methods on data quality. PCR amplification often introduces "GC-bias," leading to uneven coverage in regions with high or low GC content. In contrast, PCR-free methods provided much more uniform coverage across the genome, including critical functional regions like exons and ClinVar sites.

Impact of library preparation on coverage evenness and GC bias across different samples.

As shown in Figure 1, the variance in normalized reads relative to GC content and the percentage of the genome covered at low depths vary significantly across different sequencing protocols. PCR-free libraries typically achieved 73–74% coverage at or above 25x, whereas some PCR-based libraries dropped to as low as 46%, potentially missing critical mutations in difficult-to-sequence regions.

Sequencing Depth and the 100x Sweet Spot

While 30x coverage is a common industry standard, this study demonstrates that sensitivity improves dramatically as depth increases toward 100x. Beyond this point, the rate of discovery for new mutations begins to plateau for most samples. Crucially, the study emphasizes the need for "balanced" coverage; increasing tumor depth without a corresponding increase in the normal control depth leads to a surge in false positives, as pipelines struggle to confidently rule out germline variants.

Overlap of somatic single-base mutations (SSMs) and allele frequency distributions at varying sequencing depths.

Figure 6 illustrates the overlap of SSMs called at different balanced coverages. As depth increases, the number of "shared" mutations grows, and the allele frequency distributions (MAF) become more refined, highlighting the increased confidence and consistency in mutation calls at higher depths.

Pipeline Discordance and the Power of Consensus

The study revealed a startling lack of consistency among analytical pipelines. For the medulloblastoma sample, only a small fraction of SSMs were agreed upon by all 18 pipelines tested. This discordance reflects the artifact-prone nature of raw sequencing data and the lack of standardized filtering. However, the researchers demonstrated that using the benchmark "Gold Set" to tune filters—adjusting for factors like mapping quality or read position—could rapidly improve the performance of even poorly performing pipelines, bringing them up to the level of the field leaders.

Discussion and Future Directions

This benchmarking exercise highlights that while WGS is a powerful tool for precision oncology, its accuracy is highly dependent on technical choices. PCR-free protocols and balanced 100x coverage represent a significant step forward in diagnostic reliability. However, limitations remain: the tumors studied were highly pure (95–98%), which does not reflect the complexity of many clinical samples containing significant stromal or immune cell contamination. Future work must address low-purity samples, subclonal mutations, and larger structural variants to fully realize the potential of whole-genome sequencing in the clinic.

Summary updated

Abstract

As whole-genome sequencing for cancer genome analysis becomes a clinical tool, a full understanding of the variables affecting sequencing analysis output is required. Here using tumour-normal sample pairs from two different types of cancer, chronic lymphocytic leukaemia and medulloblastoma, we conduct a benchmarking exercise within the context of the International Cancer Genome Consortium. We compare sequencing methods, analysis pipelines and validation methods. We show that using PCR-free methods and increasing sequencing depth to ∼ 100 × shows benefits, as long as the tumour:control coverage ratio remains balanced. We observe widely varying mutation call rates and low concordance among analysis pipelines, reflecting the artefact-prone nature of the raw data and lack of standards for dealing with the artefacts. However, we show that, using the benchmark mutation set we have created, many issues are in fact easy to remedy and have an immediate positive impact on mutation detection accuracy.
BibTeX
@inproceedings{alioto2015comprehensive,
  author = {Tyler Alioto and Ivo Buchhalter and Sophia Derdak and Barbara Hutter and Matthew Eldridge and Eivind Hovig and Lawrence E. Heisler and Timothy A. Beck and Jared T. Simpson and Laurie Tonon and Anne-Sophie Sertier and Ann‐Marie Patch and Natalie Jäger and Philip Ginsbach and Ruben M. Drews and Nagarajan Paramasivam and Rolf Kabbe and Sasithorn Chotewutmontri and Nicolle Diessl and Christopher Previti and Sabine Schmidt and Benedikt Brors and Lars Feuerbach and Michael C. Heinold and Susanne Gröbner and Andrey Korshunov and Patrick Tarpey and Adam P. Butler and Jonathan Hinton and David Jones and Andrew Menzies and Keiran Raine and Rebecca Shepherd and Lucy Stebbings and Jon W. Teague and Paolo Ribeca and Francesc Castro-Giner and Sergi Beltrán and Emanuele Raineri and Marc Dabad and Simon Heath and Marta Gut and Robert E. Denroche and Nicholas J. Harding and Takafumi N. Yamaguchi and Akihiro Fujimoto and Hidewaki Nakagawa and Vı́ctor Quesada and Rafael Valdés‐Mas and Sigve Nakken and Daniel Vodák and Lawrence Bower and Andy G. Lynch and Charlotte Anderson and Nicola Waddell and John V. Pearson and Sean M. Grimmond and Myron Peto and Paul T. Spellman and Minghui He and Cyriac Kandoth and Semin Lee and John H. Zhang and Louis Létourneau and Singer Ma and Sahil Seth and David Torrents and Xi Liu and David A. Wheeler and Carlos López-Otı́n and Elı́as Campo and Peter J. Campbell and Paul C. Boutros and Xosé S. Puente and Daniela S. Gerhard and Stefan M. Pfister and John D. McPherson and Thomas J. Hudson and Matthias Schlesner and Peter Lichter and Roland Eils and David Jones and Marta Gut},
  title = {{A comprehensive assessment of somatic mutation detection in cancer using whole-genome sequencing}},
  booktitle = {Nature Communications},
  year = {2015},
  doi = {10.1038/ncomms10001},
}
Figure