Evaluation of Portable Programming Models to Accelerate LArTPC Detector Simulations

Zhihua Dong, K. Knoepfel, Meifeng Lin, B. Viren, H. YuJournal of Physics Conference Series 2023 · DOI

TL;DR

This paper tackles the challenge of accelerating complex LArTPC detector simulations, which are crucial for high-energy physics experiments like DUNE. By porting the simulation code to the Kokkos programming model, they achieve significant speedups on GPUs while maintaining portability across different hardware architectures, a critical need in the rapidly evolving computing landscape.

What Problem Does This Paper Address?

High-energy physics experiments, particularly those using Liquid Argon Time Projection Chambers (LArTPCs) like the upcoming DUNE, rely heavily on accurate simulations of detector responses. These simulations are computationally intensive and can become a major bottleneck in the analysis workflow. Traditionally, accelerating these simulations has involved specialized code for different hardware architectures (e.g., CUDA for NVIDIA GPUs, HIP for AMD GPUs). This leads to significant development effort and maintenance overhead. The core problem is the need for a portable and performant simulation solution that can run efficiently across diverse hardware platforms, from laptops to high-performance computing clusters, without requiring extensive code rewrites for each new architecture.

Methods

The authors leverage the Wire-Cell Toolkit (WCT), a C++ library for LArTPC simulations. The primary focus is on the signal simulation component, which involves a 2D convolution. To accelerate this, they adopt the Kokkos programming model. Kokkos acts as an abstraction layer, allowing developers to write C++ code that can be mapped to various backends (like OpenMP for CPUs, CUDA for NVIDIA GPUs, and HIP for AMD GPUs) during the build process.

Key methodological changes include:

  • Major overhaul of data objects: Transitioning from "arrays of structures" to "structures of arrays" to better suit parallelization and expose higher concurrency.
  • Extensions to the toolkit: Implementing a KokkosEnv context manager and special build system support.
  • Dense matrix for "Scatter Add": Using a dense matrix representation for the scatter-add operation, which surprisingly improved performance even in single-threaded execution.
  • Custom FFT Wrapper: Since Kokkos doesn't directly provide an API for optimized vendor FFT libraries, they implemented their own wrapper to interface with libraries like FFTW, cuFFT, and rocFFT.

Datasets

Not discussed in the paper. The evaluation uses simulated data generated by the Wire-Cell Toolkit itself, based on GEANT4 simulations of ionization electron groups.

Key Insights and Ideas

  • Portability is paramount: The rapid evolution of hardware architectures makes specialized, non-portable code a significant burden. Kokkos offers a promising path to address this by enabling a single codebase to target multiple platforms.
  • Data layout matters for parallelism: The refactoring of data objects from "arrays of structures" to "structures of arrays" was crucial for unlocking higher concurrency and improving parallelization performance.
  • Unexpected performance gains: Using a dense matrix for the "Scatter Add" operation, which might seem counterintuitive for sparse data, actually yielded performance improvements.
  • CPU FFT is a bottleneck: The original CPU implementation's FFT is not parallelized, leading to a relative slowdown for the OpenMP backend compared to the original CPU code when dealing with larger FFTs. This highlights a specific area for future optimization.

Key Results

  • Significant GPU Speedup: The Kokkos implementation with the CUDA backend achieved approximately 33 times faster performance compared to the original single-threaded CPU reference implementation.
  • Respectable HIP Performance: The Kokkos-HIP version (for AMD GPUs) was about 9 times faster than the CPU reference.
  • Modest OpenMP Gains: The OpenMP backend with 64 CPU threads was only marginally faster than the original CPU code, primarily due to the non-parallelized FFT.
  • Correctness Validation: Waveform comparisons between the CPU reference and Kokkos implementations (CUDA and OpenMP) showed differences at the 0.01% level, attributed to random number stream sampling differences, indicating good functional correctness.
  • Multi-process Throughput: Running multiple processes to share GPUs demonstrated that throughput increases with more processes per GPU up to a point, achieving up to 7 times more throughput with 4 GPUs and 3 times more with 1 GPU compared to a fully loaded CPU.

Limitations and Future Work

  • Kinks in Difference Plots: The authors observed some "kinks" in the difference plots between CPU-ref and Kokkos, which they are still investigating.
  • OpenMP FFT Optimization: The slowdown of the OpenMP backend due to the non-parallelized FFT is an area for ongoing optimization.
  • Under-utilized GPUs: The investigation into multi-process performance suggests that GPUs might be under-utilized with a single process, and further work is needed to understand why the throughput with 4 GPUs isn't four times that of a single GPU.
  • Further Optimization and Other Models: The authors plan to investigate further optimizations and explore other portable programming models like SYCL.

Summary updated

Abstract

Abstract The Liquid Argon Time Projection Chamber (LArTPC) technology is widely used in high energy physics experiments, including the upcoming Deep Underground Neutrino Experiment (DUNE). Accurately simulating LArTPC detector responses is essential for analysis algorithm development and physics model interpretations. Accurate LArTPC detector response simulations are computationally demanding, and can become a bottleneck in the analysis workflow. Compute devices such as General-Purpose Graphics Processing Units (GPGPUs) have the potential to substantially accelerate simulations compared to traditional CPU-only processing. The software development for these compute accelerators often carries the cost of specialized code refactorization and porting to match the target hardware architecture. With the rapid evolution and increased diversity of the computer architecture landscape, it is highly desirable to have a portable solution that also maintains reasonable performance. We report our ongoing effort in evaluating Kokkos as a basis for this portable programming model using LArTPC simulations in the context of the Wire-Cell Toolkit, a C++ library for LArTPC simulations, data analysis, reconstruction and visualization.
BibTeX
@article{dong2023evaluation,
  author = {Zhihua Dong and K. Knoepfel and Meifeng Lin and B. Viren and H. Yu},
  title = {{Evaluation of Portable Programming Models to Accelerate LArTPC Detector Simulations}},
  journal = {Journal of Physics Conference Series},
  year = {2023},
  doi = {10.1088/1742-6596/2438/1/012036},
}
Figure