TL;DR
This paper benchmarks Sentieon DNASeq, a GATK-alternative variant calling toolkit, demonstrating near-identical accuracy to GATK but with a significant speedup (up to 30x faster). It highlights Sentieon's excellent scalability and computational efficiency, making it a compelling option for clinical and research applications requiring rapid variant analysis.
What Problem Does This Paper Address?
The rapid advancement and decreasing cost of genome sequencing technologies have led to an explosion of genomic data. This creates a bottleneck in the subsequent analysis, particularly in variant calling – the process of identifying genetic differences between an individual's genome and a reference. The current industry standard, the Genome Analysis Toolkit (GATK), while accurate, can be computationally intensive, taking days or even hours for a single whole-genome sequencing (WGS) sample. This slowness hinders timely clinical diagnoses and large-scale genomic reanalyses. The paper addresses the need for faster, yet equally accurate, variant calling solutions, evaluating Sentieon DNASeq as a potential alternative.
Methods
The study employs a comparative benchmarking approach. Sentieon DNASeq's single-sample variant calling pipeline is directly compared against GATK3.8 and GATK4. The benchmarking covers several key aspects:
- Pipeline Equivalence: Sentieon DNASeq aims to replicate GATK's "best practices" algorithms and file formats, making it a direct competitor. The paper details how Sentieon's tools map to GATK's steps (e.g., alignment, sorting, deduplication, realignment, quality score recalibration, and variant calling).
- Computational Performance: This includes evaluating walltime (runtime) for the entire pipeline and individual steps, as well as assessing scalability with increasing thread counts and sequencing depth.
- Accuracy: Variant calling accuracy is assessed by comparing the output VCF files against established "truth sets" (known variant calls).
- Resource Utilization: The paper examines CPU utilization, memory usage (VmSize and VmRSS), and I/O rates (reads and writes per second) for Sentieon's tools.
A key methodological difference highlighted is Sentieon's "on-the-fly" application of Base Quality Score Recalibration (BQSR) during the variant calling step, which avoids the creation of intermediate recalibrated BAM files, contributing to its speed.
Datasets
The study utilizes three distinct datasets for evaluation:
- NA12878 WGS data (~20X depth): This dataset, downloaded from Illumina BaseSpace, was used with its corresponding Illumina Platinum variant calls as the truth set for assessing accuracy.
- NA24694 WGS data: This dataset from the Genome in a Bottle (GIAB) project was used to investigate the relationship between sequencing depth and performance. Subsets representing 25X, 50X, 75X, and 100X coverage were created.
- Synthetic WGS data (chromosomes 20-22): Generated using the
NEAT-genReadstool, this dataset allowed for precise control over introduced mutations, with a "Golden VCF" serving as the truth set. This synthetic data was crucial for comparing variant calling accuracy between Sentieon and GATK4.
Key Insights and Ideas
- Near-Identical Accuracy: The most significant finding is that Sentieon DNASeq achieves accuracy that is virtually indistinguishable from GATK, even when compared against gold-standard truth sets. This addresses a primary concern when considering alternative tools – that speed might come at the cost of accuracy.
- Exceptional Scalability: Sentieon DNASeq demonstrates near-perfect scalability with the number of threads, meaning that doubling the threads roughly halves the runtime, up to the maximum tested (40 threads). This makes it highly adaptable to various computational environments.
- "On-the-Fly" BQSR Advantage: Sentieon's implementation of BQSR during variant calling, rather than as a separate step creating intermediate files, significantly reduces I/O and speeds up the pipeline.
- Clinical Viability: The speed and accuracy demonstrated by Sentieon DNASeq open up possibilities for point-of-care patient analysis and rapid reanalysis of large historical datasets, which were previously impractical.
- Resource Efficiency: While Sentieon's optimized BWA MEM uses more memory, the overall pipeline's CPU utilization is high, suggesting it's largely CPU-bound and benefits greatly from multi-threading. Memory usage for the Haplotyper, the most memory-intensive tool, is comparable to or lower than some previous GATK benchmarks.
Key Results
- Speedup: Sentieon DNASeq completes the variant calling process from FASTQ to VCF in under 2 hours and from sorted BAM to VCF in less than half an hour for a 20X WGS sample. This represents a speedup of approximately 30x compared to GATK (up to 97% time savings).
- Accuracy:
- Sentieon vs. GATK 4: Concordance was 99.91% for synthetic data and 99.78% (with realignment) or 99.72% (without realignment) for NA12878.
- Sentieon vs. Truth Set: Achieved 92.80% concordance with the Golden VCF (synthetic data) and 94.9% with the Illumina Platinum calls (NA12878). GATK4 showed identical concordance rates on the NA12878 data.
- Scalability: The pipeline scales near-perfectly with threads up to 40, and demonstrates near-linear scaling with sequencing depth.
- BWA MEM Comparison: Sentieon's optimized BWA MEM was ~28% faster than traditional BWA MEM + samtools + NovoSort, though it used nearly double the memory.
Limitations and Future Work
- Licensing: Unlike GATK, Sentieon DNASeq is not open-source and requires a license for use.
- I/O Bottleneck Potential: While Sentieon's tools scale well on a single node, the high I/O rates during deduplication and realignment could potentially create a filesystem bottleneck when analyzing a very large number of samples (hundreds) simultaneously on a shared cluster. The authors suggest using local disks or SSDs to mitigate this.
- Future Work: The paper doesn't explicitly detail future work, but the implications of its findings point towards broader adoption of Sentieon for clinical diagnostics and large-scale genomic studies where speed is paramount. The comparison of Sentieon's optimized BWA MEM also suggests potential for further optimization in alignment.
Summary updated