A comprehensive multicenter comparison of whole genome sequencing pipelines using a uniform tumor-normal sample pair

Ivo Buchhalter, Barbara Hutter, Tyler Alioto, Timothy A. Beck, Paul C. Boutros, Benedikt Brors, Adam P. Butler, Sasithorn Chotewutmontri, Robert E. Denroche, Sophia Derdak, Nicolle Diessl, Lars Feuerbach, Akihiro Fujimoto, Susanne Gröbner, Marta Gut, Nicholas J. Harding, Michael C. Heinold, Lawrence E. Heisler, Jonathan Hinton, Natalie Jäger, David Jones, Rolf Kabbe, Andrey Korshunov, John D. McPherson, Andrew Menzies, Hidewaki Nakagawa, Christopher Previti, Keiran Raine, Paolo Ribeca, Steffen Schmidt, Rebecca Shepherd, Lucy Stebbings, Patrick Tarpey, J. Teague, Laurie Tonon, David A. Wheeler, Xi Liu, Takafumi N. Yamaguchi, Anne-Sophie Sertier, Stefan M. Pfister, Peter J. Campbell, Matthias Schlesner, Peter Lichter, Roland Eils, on behalf of the Icgc Verification and Validation Working GroupbioRxiv 2014 · DOI

TL;DR

This study benchmarks whole-genome sequencing (WGS) pipelines across multiple centers using a single tumor-normal sample pair. It reveals significant discrepancies in variant calling, highlighting the critical impact of library preparation and sequencing depth, and suggests that ~100x coverage with PCR-free libraries is a good target for robust somatic mutation detection.

What Problem Does This Paper Address?

As next-generation sequencing (NGS) moves from research into clinical settings, ensuring the reliability and comparability of results is paramount. Different laboratories use varying pipelines for WGS, from DNA library preparation to variant calling algorithms. This variability can lead to discrepancies in detected mutations, potentially impacting clinical decisions. The paper aims to understand and quantify these differences by comparing WGS pipelines from multiple independent centers using a standardized tumor-normal sample pair, identifying key factors that influence the accuracy and consistency of variant detection.

Methods

The study employed a multi-center benchmarking approach. Five independent centers (CNAG, DKFZ, OICR, RIKEN, and WTSI) processed a single tumor-blood DNA pair using their respective WGS pipelines. This involved various library preparation protocols and variant calling algorithms. To isolate the impact of analysis pipelines, a standardized algorithm (DKFZ's) was also applied to all datasets. The researchers analyzed metrics such as GC bias, coverage distribution, and the overlap of called mutations across different pipelines and coverage depths.

Datasets

The primary dataset consisted of a single tumor-blood DNA pair from a medulloblastoma patient. This pair was sequenced by five different centers, generating multiple WGS datasets. The high tumor purity (>95%) of the medulloblastoma sample was advantageous for analyzing mutation detection at various allele frequencies. The study also references a "Gold" set of curated mutations derived from a 300x WGS dataset for validation.

Key Insights and Ideas

  • Library Prep is King (or Queen): The choice of library preparation method significantly impacts sequencing quality. PCR-free libraries consistently showed more even coverage and reduced GC bias compared to those involving PCR amplification.
  • Coverage Matters, But There's a Sweet Spot: While increasing sequencing depth generally improves mutation detection, the benefit plateaus around 100x coverage. Beyond this, gains are minimal, suggesting a saturation point.
  • Balance is Key: Maintaining a balanced tumor-to-normal coverage ratio is crucial. Unbalanced coverage can lead to an increase in false positives, particularly for low-allele-frequency variants.
  • Clinical Translation Requires Standardization: For WGS to be a reliable clinical tool, pipelines must be globally compatible and comparable. This study highlights the need for rigorous benchmarking and standardization.
  • Beyond Coding Regions: Discrepancies outside of coding regions (e.g., non-coding DNA) were more substantial, suggesting that current pipelines may not fully capture the complexity of disease-associated genetic alterations in these regions.

Key Results

  • Variant Calling Discrepancies: Even with a single tumor-normal pair, analyses by individual centers revealed significant variations in called mutations. Less than 20% of called mutations were consistently identified by all five centers.
  • PCR-Free Advantage: PCR-free libraries demonstrated more even coverage and reduced GC bias, leading to better performance in detecting mutations.
  • Coverage Depth Impact: Increasing coverage from 30x to 100x significantly improved mutation detection sensitivity, with 95% of the maximum detected mutations found at 100x, compared to only 77% at 30x.
  • Artifacts are Real: A specific artifact related to base quality score adjustment was identified in one center's re-analysis, emphasizing the challenges of re-analyzing external data without known pipeline details.
  • Non-coding Regions: Differences in non-coding regions were more pronounced, indicating a need for further investigation and improved methods for analyzing these areas.

Limitations and Future Work

The authors acknowledge that each method was tested only once, limiting the assessment of reproducibility for variations and the precise contribution of individual parameters. They suggest that reporting detailed protocols and QC metrics is essential for clinical applications. Future work should involve benchmarking variant calling pipelines against publicly available datasets of validated mutations, including the "Gold" set generated in this study. The study also implies a need for further investigation into the detection of alterations in non-coding regions.

Summary updated

Abstract

Abstract As next-generation sequencing becomes a clinical tool, a full understanding of the variables affecting sequencing analysis output is required. Through the International Cancer Genome Consortium (ICGC), we compared sequencing pipelines at five independent centers (CNAG, DKFZ, OICR, RIKEN and WTSI) using a single tumor-blood DNA pair. Analyses by each center and with one standardized algorithm revealed significant discrepancies. Although most pipelines performed well for coding mutations, library preparation methods and sequencing coverage metrics clearly influenced downstream results. PCR-free methods showed reduced GC-bias and more even coverage. Increasing sequencing depth to ∼100x (two- to three-fold higher than current standards) showed a benefit, as long as the tumor:control coverage ratio remained balanced. To become part of routine clinical care, high-throughput sequencing must be globally compatible and comparable. This benchmarking exercise has highlighted several fundamental parameters to consider in this regard, which will allow for better optimization and planning of both basic and translational studies.
BibTeX
@misc{buchhalter2014comprehensive,
  author = {Ivo Buchhalter and Barbara Hutter and Tyler Alioto and Timothy A. Beck and Paul C. Boutros and Benedikt Brors and Adam P. Butler and Sasithorn Chotewutmontri and Robert E. Denroche and Sophia Derdak and Nicolle Diessl and Lars Feuerbach and Akihiro Fujimoto and Susanne Gröbner and Marta Gut and Nicholas J. Harding and Michael C. Heinold and Lawrence E. Heisler and Jonathan Hinton and Natalie Jäger and David Jones and Rolf Kabbe and Andrey Korshunov and John D. McPherson and Andrew Menzies and Hidewaki Nakagawa and Christopher Previti and Keiran Raine and Paolo Ribeca and Steffen Schmidt and Rebecca Shepherd and Lucy Stebbings and Patrick Tarpey and J. Teague and Laurie Tonon and David A. Wheeler and Xi Liu and Takafumi N. Yamaguchi and Anne-Sophie Sertier and Stefan M. Pfister and Peter J. Campbell and Matthias Schlesner and Peter Lichter and Roland Eils and Marta Gut and David Jones and on behalf of the ICGC Verification and Validation Working Group},
  title = {{A comprehensive multicenter comparison of whole genome sequencing pipelines using a uniform tumor-normal sample pair}},
  year = {2014},
  doi = {10.1101/013177},
  howpublished = {bioRxiv},
}
Figure