TL;DR
Calling somatic mutations in cancer genomes is surprisingly challenging, even with advanced sequencing. This paper benchmarks various methods using real cancer data, revealing significant variability and identifying key issues that hinder clinical adoption. The findings highlight the need for standardized best practices to ensure reliable and reproducible mutation detection.
What Problem Does This Paper Address?
The advent of next-generation sequencing (NGS) has revolutionized cancer genomics, enabling large-scale projects like the International Cancer Genome Consortium (ICGC) to systematically scan cancer genomes for mutations. However, the field is still grappling with how to reliably identify these "somatic mutations" – changes that occur in tumor cells but not in normal cells. This is crucial for understanding cancer development and for potential clinical applications. The problem is that experimental and analytical procedures are constantly evolving, leading to a lack of standardized best practices. This results in widely varying mutation call rates and low concordance among different analysis pipelines, making it difficult to trust the findings. The paper aims to address this by conducting a comprehensive benchmark to identify and resolve issues in somatic mutation calling.
Methods
The core of this study is a large-scale benchmarking exercise involving multiple research groups within the ICGC. Two main benchmarks were conducted:
- BM1.1 (Chronic Lymphocytic Leukemia - CLL): Participants were provided with whole-genome sequencing (WGS) data from a CLL tumor and its matched normal sample. They were asked to submit their calls for somatic single-base mutations (SSMs) and somatic insertion/deletion mutations (SIMs), as well as structural variants.
- BM1.2 (Medulloblastoma - MB): Building on the experience from BM1.1, this benchmark used higher quality sequencing data and refined procedures.
To evaluate the submissions, a "Gold Set" of verified mutations was created for each cancer type. This involved:
- Orthogonal Verification: Using independent target capture experiments and sequencing platforms (e.g., Illumina MiSeq, IonTorrent PGM) to re-sequence regions identified by multiple callers or private calls.
- Manual Review: Expert reviewers visually inspected mutation calls and their alignments.
- Consensus Calling: For BM1.2, a consensus approach was used, requiring agreement among multiple pipelines to define a "true" mutation.
The study also delved into specific components of the analysis pipeline, including different mappers (alignment tools), mutation callers, and reference genome builds, to understand their impact on results.
Datasets
The study utilized two primary datasets:
- Chronic Lymphocytic Leukemia (CLL) dataset: Whole-genome sequencing data from a tumor-normal pair.
- Medulloblastoma (MB) dataset: Whole-genome sequencing data from a tumor-normal pair.
These datasets were used to create the CLL Gold Set and the MB Gold Set, which served as the ground truth for evaluating the submitted mutation calls. These Gold Sets represent a novel benchmark resource for the community.
Key Insights and Ideas
- Somatic mutation calling is far from solved: Despite advances in sequencing technology, there's a significant lack of agreement among different analysis pipelines, leading to widely varying results. This suggests that "somatic mutation calling remains an unsolved problem."
- "Artefacts" are a major hurdle: The study highlights that a lack of consensus on what constitutes an artefact (e.g., sequencing errors, alignment issues) and how to handle them is a significant contributor to discordant calls.
- Pipeline components matter, but how they're used matters more: While different mappers and callers have their own biases, the way these tools are integrated and configured within a pipeline is critical. No single tool or combination emerged as universally superior.
- Genomic context is important: Certain genomic regions (e.g., repetitive sequences, centromeric regions) are more prone to false positives or false negatives, suggesting that alignment and variant calling algorithms need to be more sophisticated in handling these areas.
- Real data is essential for benchmarking: The study emphasizes the value of using real biological samples rather than simulated data for benchmarks, as it better reflects the complexities and artefacts encountered in actual research.
Key Results
- Wide variation in call rates: For SSMs, the number of calls submitted by different groups ranged from 467 to 3159 for the CLL benchmark and 801 to 8876 for the MB benchmark. Similar variability was observed for SIMs.
- Low concordance: The agreement between different submissions and the Gold Set was often low, indicating that many calls were either false positives or false negatives.
- Specific challenges identified:
- SSMs: Low coverage in the tumor and aberrant coverage of tumor/normal samples were identified as major challenges.
- SIMs: Repetitive features (e.g., interspersed repeats, simple repeats, homopolymers) tended to dominate issues for SIM calling.
- Pipeline combinations can improve performance: Pipelines that combined multiple mutation callers (e.g., MuTect and Strelka) showed more balanced performance in terms of precision and recall.
- Mapper and caller biases: Each tested mapper and caller exhibited specific biases that could lead to a significant number of false positive calls if not addressed.
Limitations and Future Work
- Limitations:
- The study acknowledges that false negative calls cannot be assessed exhaustively, as verification methods are better at confirming the presence of a mutation than its absence.
- The complexity of the analysis pipelines made it impossible to test every single component and parameter setting.
- The study focused on two cancer types, and findings may not be universally applicable to all cancers.
- Future Work:
- The authors suggest that further benchmarks are needed to address more difficult mutational features, such as low sample purity, subclonality, structural rearrangements, and karyotype aberrations.
- Developing better tools for these complex features is crucial for making cancer sequencing adoptable for clinical use.
- The study provides the sequencing reads and Gold Sets for BM1.2 to the research community for further benchmarking of their own pipelines.
- General advice is given to start with a basic understanding of the tumor and tumor type being analyzed to adapt downstream analysis.
Summary updated