A comparison of tools for copy-number variation detection in germline whole exome and whole genome sequencing data

Miglė Gabrielaitė, Mathias H. Torp, Malthe Sebro Rasmussen, Sergio Andreu‐Sánchez, Filipe Garrett Vieira, Christina B. Pedersen, Savvas Kinalis, Majbritt Busk Madsen, Christina W. Yde, Lars Rønn Olsen, Rasmus L. Marvig, Oľga Østrup, Maria Rossing, Finn Cilius Nielsen, Ole Winther, Frederik Otzen BaggerbioRxiv 2021 · DOI

TL;DR

Copy number variations (CNVs) are crucial for understanding diseases, but detecting them accurately from sequencing data is challenging. This study benchmarks 11 popular CNV calling tools using both whole-exome (WES) and whole-genome (WGS) sequencing data, revealing significant performance differences and suggesting that combining top-performing tools like GATK gCNV, Lumpy, DELLY, and cn.MOPS is key for reliable detection.

What Problem Does This Paper Address?

Copy number variations (CNVs) are structural changes in the genome where segments are duplicated or deleted. These variations play a significant role in various diseases, including cancer and neurodevelopmental disorders, and can affect gene expression. While array-based methods have been the clinical standard for CNV detection, next-generation sequencing (NGS) technologies like whole-exome sequencing (WES) and whole-genome sequencing (WGS) offer the potential for more comprehensive genomic profiling. However, accurately calling CNVs from NGS data is difficult due to challenges like variable sequencing coverage, alignment biases, and the sheer number of available tools with varying performance. Researchers need to know which tools are reliable for clinical applications and what sequencing protocols are best suited for CNV detection, but a lack of standardized benchmarks and reference datasets has made this difficult.

Methods

The researchers reviewed 50 CNV calling tools and selected 11 for a comprehensive benchmark. They employed a multi-pronged approach:

  • Benchmarking Framework: They used a reference cohort of 39 WGS samples and compared the results against the established clinical standard, SNP-array based CNV calling.
  • WES vs. WGS Comparison: For nine of these samples, they also performed WES to assess the impact of sequencing protocol on CNV calling performance.
  • Gold Standard and Validation: They included the well-characterized NA12878 sample as a gold standard and tested 12 samples with CNVs confirmed by multiplex ligation-dependent probe amplification (MLPA) for further validation.
  • Tool Evaluation: The 11 selected tools were run using their default parameters and author recommendations. Performance was assessed based on recall (ability to find true CNVs) and precision (proportion of called CNVs that are true positives).

Datasets

The study utilized the following datasets for benchmarking and evaluation:

  • NA12878 Gold Standard: A well-characterized sample with known CNVs, used as a gold standard.
  • In-house Cohort (GB01-GB38): 39 WGS samples and 8 WES samples from an in-house cohort.
  • Reference CNVs:
    • For NA12878: CNVs from Haraksingh et al., 2017, and in-house WGS data.
  • For GB01-GB38: CNVs derived from CytoScan HD SNP-array.
  • For specific validation: CNVs confirmed by MLPA.

The study effectively created a unique reference cohort for benchmarking, combining established gold standards with in-house data and array-based calls.

Key Insights and Ideas

  • Tool Performance Varies Wildly: There's no single "best" tool; performance differs significantly across datasets and metrics (recall vs. precision).
  • WGS Outperforms WES for CNV Calling: Even tools designed for WES struggled with WES data, suggesting WGS is the preferred method for comprehensive CNV detection.
  • Overfitting is a Concern: Tools that performed exceptionally well on the NA12878 gold standard sample showed discrepancies on the in-house cohort, hinting at potential overfitting during tool development.
  • Recall vs. Precision Trade-off: Tools that identified more CNVs (higher recall) often had lower precision, meaning they generated more false positives.
  • Filtering Doesn't Always Help: Applying confidence cutoffs to filter CNV calls didn't consistently improve precision and sometimes reduced recall.
  • Combining Tools is Crucial: No single tool is perfect. The authors recommend combining the top-performing tools for a more balanced and accurate CNV detection.

Key Results

  • Recall Leaders: GATK gCNV, Lumpy, DELLY, and cn.MOPS demonstrated the highest recall across both WES and WGS data.
  • Precision Challenges: Precision remained a significant challenge for most tools, with maximum precision rarely exceeding 66.7% for WGS.
  • WES Limitations: Most tools performed poorly on WES data, with some missing all MLPA-confirmed CNVs.
  • WGS Strengths: WGS data generally yielded better results, with tools like DELLY and GATK gCNV calling a large number of CNVs.
  • Short CNVs are Tricky: A significant portion of called CNVs were shorter than 1,000 bp, and these were often called by only one tool, making them harder to validate.
  • Recommended Combination: For a balanced CNV recall and precision, the authors suggest combining GATK gCNV, Lumpy, DELLY, and cn.MOPS.

Limitations and Future Work

  • Limited Tool Set: While 11 tools were benchmarked, this is still a subset of all available CNV calling tools.
  • "Truth" Set Limitations: The study acknowledges that even array-based calls (used as a reference) have limitations and may not capture all short CNVs. The NA12878 gold standard, while extensive, might have contributed to potential overfitting in some tools.
  • Well-defined CNV Sets Needed: The authors emphasize the need for more large, well-annotated datasets with CNV calls from multiple platforms to improve tool development and evaluation.
  • Future Directions:
    • Improved Annotation: Better annotation of CNVs is crucial for understanding their functional impact and prioritizing calls.
  • Leveraging Other Data: Integrating other genomic data (e.g., RNA-seq) and knowledge of genetic sites could aid in CNV prioritization.
  • Background Panels: Creating databases of common CNVs or false positives can help filter results in a clinical setting.
  • Long-read Sequencing: Future advancements may involve combining long- and short-read sequencing for more accurate CNV detection and breakpoint resolution.
  • Clinical Application: For clinical use, a hypothesis-based approach combined with manual inspection and validation (MLPA, arrays) is recommended.

Summary updated

Abstract

Abstract Background Copy-number variations (CNVs) have important clinical implications for several diseases and cancers. The clinically relevant CNVs are hard to detect because CNVs are common structural variations that define large parts of the normal human genome. CNV calling from short-read sequencing data has the potential to leverage available cohort studies and allow full genomic profiling in the clinic without the need for additional data modalities. Questions regarding performance of CNV calling tools for clinical use and suitable sequencing protocols remain poorly addressed, mainly because of the lack of good reference data sets. Methods We reviewed 50 popular CNV calling tools and included 11 tools for benchmarking in a unique reference cohort encompassing 39 whole genome sequencing (WGS) samples paired with analysis by the current clinical standard —SNP-array based CNV calling. Additionally, for nine of these samples we performed whole exome sequencing (WES) performed, in order to address the effect of sequencing protocol on CNV calling. Furthermore, we included Gold Standard reference sample NA12878, and tested 12 samples with CNVs confirmed by multiplex ligation-dependent probe amplification (MLPA). Results Tool performance varied greatly in the number of called CNVs and bias for CNV lengths. Some tools had near-perfect recall of CNVs from arrays for some samples, but poor precision. Filtering output by CNV ranks from tools did not salvage precision. Several tools had better performance patterns for NA12878, and we hypothesize that this is the result of overfitting during the tool development. Conclusions We suggest combining tools with the best recall: GATK gCNV, Lumpy, DELLY, and cn.MOPS. These tools also capture different CNVs. Further improvements in precision requires additional development of tools, reference data sets, and annotation of CNVs, potentially assisted by the use of background panels for filtering of frequently called variants.
BibTeX
@misc{gabrielaite2021comparison,
  author = {Miglė Gabrielaitė and Mathias H. Torp and Malthe Sebro Rasmussen and Sergio Andreu‐Sánchez and Filipe Garrett Vieira and Christina B. Pedersen and Savvas Kinalis and Majbritt Busk Madsen and Christina W. Yde and Lars Rønn Olsen and Rasmus L. Marvig and Oľga Østrup and Maria Rossing and Finn Cilius Nielsen and Ole Winther and Frederik Otzen Bagger},
  title = {{A comparison of tools for copy-number variation detection in germline whole exome and whole genome sequencing data}},
  year = {2021},
  doi = {10.1101/2021.04.30.442110},
  howpublished = {bioRxiv},
}
Figure