Recommendations for performance optimizations when using GATK3.8 and GATK4

Jacob R Heldenbrand, Saurabh Baheti, Matthew A. Bockol, Travis M. Drucker, Steven N. Hart, Matthew E. Hudson, Ravishankar K. Iyer, Michael Kalmbach, Katherine Irene Kendig, Eric W. Klee, Nathan R Mattson, Eric D. Wieben, Mathieu Wiepert, Derek E. Wildman, Liudmila S. MainzerBMC Bioinformatics 2019 · DOI

Abstract

In time-sensitive situations, such as when a patient has a critical or rapidly developing condition, it is essential to minimize the time required to process a single genomic sample. In these cases, we recommend using GATK3.8 by splitting the sample into chunks and computing across multiple nodes. This approach achieves a walltime of 3.4 hours at a cost of $41.60 using four c5.18xlarge Amazon Cloud instances. Conversely, for routine analyses or large-scale population studies where cost-effectiveness is paramount, we recommend GATK4. By running multiple samples on a single node, the total walltime for 40 samples is approximately 34.1 hours, resulting in a throughput of 1.18 samples per hour at a cost of only $2.60 per sample.

Introduction

Genomic variant calling—the identification of genetic differences between an individual's DNA and a reference genome—is the industry standard for clinical and research applications. However, as sequencing technology has advanced, the computational processing of this data has become a significant bottleneck. The Genome Analysis Toolkit (GATK) is the gold standard for this process, but its performance is highly dependent on software parameters, threading models, and hardware configurations.

With the transition from GATK3.8 to GATK4, researchers have faced challenges in optimizing their workflows. This paper provides a roadmap for tuning these versions based on two primary objectives: absolute speed for urgent clinical diagnostics and cost-effective throughput for large-scale studies.

Methodology

The study involved a comprehensive scalability analysis conducted on Intel Skylake-based high-performance computing (HPC) nodes. The researchers evaluated several optimization vectors:

  • Parameter Tuning: Adjusting internal threading options such as -nct and -nt.
  • Java Memory Management: Testing Parallel Garbage Collection (PGC) settings to optimize memory usage in GATK4.
  • I/O Optimization: Benchmarking asynchronous read/write options for Samtools and Tribble file formats.
  • Parallelization: Implementing data-level parallelization by splitting the genome into chromosomal chunks for multi-node processing.
  • Cost Analysis: Translating performance metrics into actual dollar costs using Amazon Web Services (AWS) pricing models.

The benchmarks utilized the NA12878 whole-genome sequencing dataset (~20X depth) mapped to the hg38 human reference genome.

Results and Discussion

The Threading Trap

A critical finding of this study is that increasing thread counts does not linearly improve performance across all GATK tools. In GATK3.8, tools like PrintReads exhibit degraded performance when exceeding three threads due to I/O and memory contention.

Fold change in performance across different GATK tools as thread count increases, showing the stagnation of PrintReads compared to other tools.

As shown in the figure above, while BaseRecalibrator and HaplotypeCaller show initial scaling, the gains diminish significantly at higher thread counts. This "threading trap" necessitates a more nuanced approach to parallelization.

Optimization and Walltime

Systematic parameter tuning resulted in execution time reductions of 29.3% for GATK3.8 and 16.9% for GATK4. However, the study found that for certain tools, walltime actually increases if too many threads are allocated to a single task.

Walltime vs. thread count, illustrating how excessive threading can lead to increased execution times.

To overcome these local bottlenecks, the authors recommend "across-node" parallelization for urgent cases. By splitting a single sample into chunks and processing them across four physical servers, the total walltime can be reduced to 3.4 hours.

GATK4 for High-Throughput Efficiency

While GATK3.8 excels in single-sample speed when heavily parallelized across nodes, GATK4 is designed for throughput. GATK4 tools are largely single-threaded, which allows users to "pack" multiple samples onto a single multi-core node. This strategy minimizes resource conflict and maximizes hardware utilization, reducing the cost per sample to $2.60—a fraction of the cost of the high-speed GATK3.8 configuration.

Conclusion

Choosing between GATK3.8 and GATK4 depends on the specific requirements of the project. For urgent clinical needs, GATK3.8 with multi-node splitting offers the fastest turnaround. For large-scale research, GATK4 provides superior cost-efficiency. Future work will evaluate the performance of GATK4’s Spark-based tools, which were in beta during this study, to determine if they can bridge the gap between speed and throughput.

Summary updated

BibTeX
@article{heldenbrand2019recommendations,
  author = {Jacob R Heldenbrand and Saurabh Baheti and Matthew A. Bockol and Travis M. Drucker and Steven N. Hart and Matthew E. Hudson and Ravishankar K. Iyer and Michael Kalmbach and Katherine Irene Kendig and Eric W. Klee and Nathan R Mattson and Eric D. Wieben and Mathieu Wiepert and Derek E. Wildman and Liudmila S. Mainzer},
  title = {{Recommendations for performance optimizations when using GATK3.8 and GATK4}},
  journal = {BMC Bioinformatics},
  year = {2019},
  doi = {10.1186/s12859-019-3169-7},
}
Figure