A Comprehensive Assessment of Somatic Mutation Calling in Cancer Genomes

Tyler Alioto, Sophia Derdak, Timothy A. Beck, Paul C. Boutros, Lawrence Bower, Ivo Buchhalter, Matthew Eldridge, Nicholas J. Harding, Lawrence E. Heisler, Eivind Hovig, David Jones, Andy G. Lynch, Sigve Nakken, Paolo Ribeca, Anne-Sophie Sertier, Jared T. Simpson, Paul T. Spellman, Patrick Tarpey, Laurie Tonon, Daniel Vodák, Takafumi N. Yamaguchi, Sergi Beltrán, Marc Dabad, Robert E. Denroche, Philip Ginsbach, Simon Heath, Emanuele Raineri, Charlotte Anderson, Benedikt Brors, Ruben M. Drews, Roland Eils, Akihiro Fujimoto, Francesc Castro-Giner, Minghui He, Pablo H. Hennings-Yeomans, Barbara Hutter, Natalie Jäger, Rolf Kabbe, Cyriac Kandoth, Semin Lee, Louis Létourneau, Singer Ma, Hidewaki Nakagawa, Nagarajan Paramasivam, Anne-Marie Patch, Myron Peto, Matthias Schlesner, Sahil Seth, David Torrents, David A. Wheeler, Xi Liu, John H. Zhang, Daniela S. Gerhard, Vı́ctor Quesada, Rafael Valdés‐Mas, Thomas J. Hudson, John D. McPherson, Xosé S. Puente, Marta Gut, Ivo G. GutbioRxiv 2014 · DOI

TL;DR

Calling somatic mutations in cancer genomes is surprisingly challenging, even with advanced sequencing. This paper benchmarks various methods using real cancer data, revealing significant variability and identifying key issues that hinder clinical adoption. The findings highlight the need for standardized best practices to ensure reliable and reproducible mutation detection.

What Problem Does This Paper Address?

The advent of next-generation sequencing (NGS) has revolutionized cancer genomics, enabling large-scale projects like the International Cancer Genome Consortium (ICGC) to systematically scan cancer genomes for mutations. However, the field is still grappling with how to reliably identify these "somatic mutations" – changes that occur in tumor cells but not in normal cells. This is crucial for understanding cancer development and for potential clinical applications. The problem is that experimental and analytical procedures are constantly evolving, leading to a lack of standardized best practices. This results in widely varying mutation call rates and low concordance among different analysis pipelines, making it difficult to trust the findings. The paper aims to address this by conducting a comprehensive benchmark to identify and resolve issues in somatic mutation calling.

Methods

The core of this study is a large-scale benchmarking exercise involving multiple research groups within the ICGC. Two main benchmarks were conducted:

  • BM1.1 (Chronic Lymphocytic Leukemia - CLL): Participants were provided with whole-genome sequencing (WGS) data from a CLL tumor and its matched normal sample. They were asked to submit their calls for somatic single-base mutations (SSMs) and somatic insertion/deletion mutations (SIMs), as well as structural variants.
  • BM1.2 (Medulloblastoma - MB): Building on the experience from BM1.1, this benchmark used higher quality sequencing data and refined procedures.

To evaluate the submissions, a "Gold Set" of verified mutations was created for each cancer type. This involved:

  • Orthogonal Verification: Using independent target capture experiments and sequencing platforms (e.g., Illumina MiSeq, IonTorrent PGM) to re-sequence regions identified by multiple callers or private calls.
  • Manual Review: Expert reviewers visually inspected mutation calls and their alignments.
  • Consensus Calling: For BM1.2, a consensus approach was used, requiring agreement among multiple pipelines to define a "true" mutation.

The study also delved into specific components of the analysis pipeline, including different mappers (alignment tools), mutation callers, and reference genome builds, to understand their impact on results.

Datasets

The study utilized two primary datasets:

  • Chronic Lymphocytic Leukemia (CLL) dataset: Whole-genome sequencing data from a tumor-normal pair.
  • Medulloblastoma (MB) dataset: Whole-genome sequencing data from a tumor-normal pair.

These datasets were used to create the CLL Gold Set and the MB Gold Set, which served as the ground truth for evaluating the submitted mutation calls. These Gold Sets represent a novel benchmark resource for the community.

Key Insights and Ideas

  • Somatic mutation calling is far from solved: Despite advances in sequencing technology, there's a significant lack of agreement among different analysis pipelines, leading to widely varying results. This suggests that "somatic mutation calling remains an unsolved problem."
  • "Artefacts" are a major hurdle: The study highlights that a lack of consensus on what constitutes an artefact (e.g., sequencing errors, alignment issues) and how to handle them is a significant contributor to discordant calls.
  • Pipeline components matter, but how they're used matters more: While different mappers and callers have their own biases, the way these tools are integrated and configured within a pipeline is critical. No single tool or combination emerged as universally superior.
  • Genomic context is important: Certain genomic regions (e.g., repetitive sequences, centromeric regions) are more prone to false positives or false negatives, suggesting that alignment and variant calling algorithms need to be more sophisticated in handling these areas.
  • Real data is essential for benchmarking: The study emphasizes the value of using real biological samples rather than simulated data for benchmarks, as it better reflects the complexities and artefacts encountered in actual research.

Key Results

  • Wide variation in call rates: For SSMs, the number of calls submitted by different groups ranged from 467 to 3159 for the CLL benchmark and 801 to 8876 for the MB benchmark. Similar variability was observed for SIMs.
  • Low concordance: The agreement between different submissions and the Gold Set was often low, indicating that many calls were either false positives or false negatives.
  • Specific challenges identified:
    • SSMs: Low coverage in the tumor and aberrant coverage of tumor/normal samples were identified as major challenges.
  • SIMs: Repetitive features (e.g., interspersed repeats, simple repeats, homopolymers) tended to dominate issues for SIM calling.
  • Pipeline combinations can improve performance: Pipelines that combined multiple mutation callers (e.g., MuTect and Strelka) showed more balanced performance in terms of precision and recall.
  • Mapper and caller biases: Each tested mapper and caller exhibited specific biases that could lead to a significant number of false positive calls if not addressed.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that false negative calls cannot be assessed exhaustively, as verification methods are better at confirming the presence of a mutation than its absence.
  • The complexity of the analysis pipelines made it impossible to test every single component and parameter setting.
  • The study focused on two cancer types, and findings may not be universally applicable to all cancers.
  • Future Work:
    • The authors suggest that further benchmarks are needed to address more difficult mutational features, such as low sample purity, subclonality, structural rearrangements, and karyotype aberrations.
  • Developing better tools for these complex features is crucial for making cancer sequencing adoptable for clinical use.
  • The study provides the sequencing reads and Gold Sets for BM1.2 to the research community for further benchmarking of their own pipelines.
  • General advice is given to start with a basic understanding of the tumor and tumor type being analyzed to adapt downstream analysis.

Summary updated

Abstract

Abstract The emergence of next generation DNA sequencing technology is enabling high-resolution cancer genome analysis. Large-scale projects like the International Cancer Genome Consortium (ICGC) are systematically scanning cancer genomes to identify recurrent somatic mutations. Second generation DNA sequencing, however, is still an evolving technology and procedures, both experimental and analytical, are constantly changing. Thus the research community is still defining a set of best practices for cancer genome data analysis, with no single protocol emerging to fulfil this role. Here we describe an extensive benchmark exercise to identify and resolve issues of somatic mutation calling. Whole genome sequence datasets comprising tumor-normal pairs from two different types of cancer, chronic lymphocytic leukaemia and medulloblastoma, were shared within the ICGC and submissions of somatic mutation calls were compared to verified mutations and to each other. Varying strategies to call mutations, incomplete awareness of sources of artefacts, and even lack of agreement on what constitutes an artefact or real mutation manifested in widely varying mutation call rates and somewhat low concordance among submissions. We conclude that somatic mutation calling remains an unsolved problem. However, we have identified many issues that are easy to remedy that are presented here. Our study highlights critical issues that need to be addressed before this valuable technology can be routinely used to inform clinical decision-making. Abbreviations and Definitions SSM Somatic Single-base Mutations or Simple Somatic Mutations, refers to a somatic single base change SIM Somatic Insertion/deletion Mutation CNV Copy Number Variant SV Structural Variant SNP Single Nucleotide Polymorphisms, refers to a single base variable position in the germline with a frequency of > 1% in the general population CLL Chronic Lymphocytic Leukaemia MB Medulloblastoma ICGC International Cancer Genome Consortium BM Benchmark aligner = mapper, these terms are used interchangeably
BibTeX
@misc{alioto2014comprehensive,
  author = {Tyler Alioto and Sophia Derdak and Timothy A. Beck and Paul C. Boutros and Lawrence Bower and Ivo Buchhalter and Matthew Eldridge and Nicholas J. Harding and Lawrence E. Heisler and Eivind Hovig and David Jones and Andy G. Lynch and Sigve Nakken and Paolo Ribeca and Anne-Sophie Sertier and Jared T. Simpson and Paul T. Spellman and Patrick Tarpey and Laurie Tonon and Daniel Vodák and Takafumi N. Yamaguchi and Sergi Beltrán and Marc Dabad and Robert E. Denroche and Philip Ginsbach and Simon Heath and Emanuele Raineri and Charlotte Anderson and Benedikt Brors and Ruben M. Drews and Roland Eils and Akihiro Fujimoto and Francesc Castro-Giner and Minghui He and Pablo H. Hennings-Yeomans and Barbara Hutter and Natalie Jäger and Rolf Kabbe and Cyriac Kandoth and Semin Lee and Louis Létourneau and Singer Ma and Hidewaki Nakagawa and Nagarajan Paramasivam and Anne-Marie Patch and Myron Peto and Matthias Schlesner and Sahil Seth and David Torrents and David A. Wheeler and Xi Liu and John H. Zhang and Daniela S. Gerhard and Vı́ctor Quesada and Rafael Valdés‐Mas and Thomas J. Hudson and John D. McPherson and Xosé S. Puente and Marta Gut and Ivo G. Gut},
  title = {{A Comprehensive Assessment of Somatic Mutation Calling in Cancer Genomes}},
  year = {2014},
  doi = {10.1101/012997},
  howpublished = {bioRxiv},
}

Loading related content...

Figure