TL;DR
Copy number variations (CNVs) are crucial for understanding diseases, but detecting them accurately from sequencing data is challenging. This study benchmarks 11 popular CNV calling tools using both whole-exome (WES) and whole-genome (WGS) sequencing data, revealing significant performance differences and suggesting that combining top-performing tools like GATK gCNV, Lumpy, DELLY, and cn.MOPS is key for reliable detection.
What Problem Does This Paper Address?
Copy number variations (CNVs) are structural changes in the genome where segments are duplicated or deleted. These variations play a significant role in various diseases, including cancer and neurodevelopmental disorders, and can affect gene expression. While array-based methods have been the clinical standard for CNV detection, next-generation sequencing (NGS) technologies like whole-exome sequencing (WES) and whole-genome sequencing (WGS) offer the potential for more comprehensive genomic profiling. However, accurately calling CNVs from NGS data is difficult due to challenges like variable sequencing coverage, alignment biases, and the sheer number of available tools with varying performance. Researchers need to know which tools are reliable for clinical applications and what sequencing protocols are best suited for CNV detection, but a lack of standardized benchmarks and reference datasets has made this difficult.
Methods
The researchers reviewed 50 CNV calling tools and selected 11 for a comprehensive benchmark. They employed a multi-pronged approach:
- Benchmarking Framework: They used a reference cohort of 39 WGS samples and compared the results against the established clinical standard, SNP-array based CNV calling.
- WES vs. WGS Comparison: For nine of these samples, they also performed WES to assess the impact of sequencing protocol on CNV calling performance.
- Gold Standard and Validation: They included the well-characterized NA12878 sample as a gold standard and tested 12 samples with CNVs confirmed by multiplex ligation-dependent probe amplification (MLPA) for further validation.
- Tool Evaluation: The 11 selected tools were run using their default parameters and author recommendations. Performance was assessed based on recall (ability to find true CNVs) and precision (proportion of called CNVs that are true positives).
Datasets
The study utilized the following datasets for benchmarking and evaluation:
- NA12878 Gold Standard: A well-characterized sample with known CNVs, used as a gold standard.
- In-house Cohort (GB01-GB38): 39 WGS samples and 8 WES samples from an in-house cohort.
- Reference CNVs:
- For NA12878: CNVs from Haraksingh et al., 2017, and in-house WGS data.
- For GB01-GB38: CNVs derived from CytoScan HD SNP-array.
- For specific validation: CNVs confirmed by MLPA.
The study effectively created a unique reference cohort for benchmarking, combining established gold standards with in-house data and array-based calls.
Key Insights and Ideas
- Tool Performance Varies Wildly: There's no single "best" tool; performance differs significantly across datasets and metrics (recall vs. precision).
- WGS Outperforms WES for CNV Calling: Even tools designed for WES struggled with WES data, suggesting WGS is the preferred method for comprehensive CNV detection.
- Overfitting is a Concern: Tools that performed exceptionally well on the NA12878 gold standard sample showed discrepancies on the in-house cohort, hinting at potential overfitting during tool development.
- Recall vs. Precision Trade-off: Tools that identified more CNVs (higher recall) often had lower precision, meaning they generated more false positives.
- Filtering Doesn't Always Help: Applying confidence cutoffs to filter CNV calls didn't consistently improve precision and sometimes reduced recall.
- Combining Tools is Crucial: No single tool is perfect. The authors recommend combining the top-performing tools for a more balanced and accurate CNV detection.
Key Results
- Recall Leaders: GATK gCNV, Lumpy, DELLY, and cn.MOPS demonstrated the highest recall across both WES and WGS data.
- Precision Challenges: Precision remained a significant challenge for most tools, with maximum precision rarely exceeding 66.7% for WGS.
- WES Limitations: Most tools performed poorly on WES data, with some missing all MLPA-confirmed CNVs.
- WGS Strengths: WGS data generally yielded better results, with tools like DELLY and GATK gCNV calling a large number of CNVs.
- Short CNVs are Tricky: A significant portion of called CNVs were shorter than 1,000 bp, and these were often called by only one tool, making them harder to validate.
- Recommended Combination: For a balanced CNV recall and precision, the authors suggest combining GATK gCNV, Lumpy, DELLY, and cn.MOPS.
Limitations and Future Work
- Limited Tool Set: While 11 tools were benchmarked, this is still a subset of all available CNV calling tools.
- "Truth" Set Limitations: The study acknowledges that even array-based calls (used as a reference) have limitations and may not capture all short CNVs. The NA12878 gold standard, while extensive, might have contributed to potential overfitting in some tools.
- Well-defined CNV Sets Needed: The authors emphasize the need for more large, well-annotated datasets with CNV calls from multiple platforms to improve tool development and evaluation.
- Future Directions:
- Improved Annotation: Better annotation of CNVs is crucial for understanding their functional impact and prioritizing calls.
- Leveraging Other Data: Integrating other genomic data (e.g., RNA-seq) and knowledge of genetic sites could aid in CNV prioritization.
- Background Panels: Creating databases of common CNVs or false positives can help filter results in a clinical setting.
- Long-read Sequencing: Future advancements may involve combining long- and short-read sequencing for more accurate CNV detection and breakpoint resolution.
- Clinical Application: For clinical use, a hypothesis-based approach combined with manual inspection and validation (MLPA, arrays) is recommended.
Summary updated