DUNE Software and High Performance Computing

B. T. Fleming, K. Knoepfel, Meifeng Lin, X. Qian, Yihui Ren, Brett Viren, H. Wei, Shinjae Yoo, Haiwang YuarXiv 2022 · DOI · arXiv · OpenAlex

Abstract

DUNE, like other HEP experiments, faces a challenge related to matching execution patterns of our production simulation and data processing software to the limitations imposed by modern high-performance computing facilities. In order to efficiently exploit these new architectures, particularly those with high CPU core counts and GPU accelerators, our existing software execution models require adaptation. In addition, the large size of individual units of raw data from the far detector modules pose an additional challenge somewhat unique to DUNE. Here we describe some of these problems and how we begin to solve them today with existing software frameworks and toolkits. We also describe ways we may leverage these existing software architectures to attack remaining problems going forward. This whitepaper is a contribution to the Computational Frontier of Snowmass21.

Introduction

The Deep Underground Neutrino Experiment (DUNE) is transitioning its software from traditional, single-threaded event processing to a highly parallel, graph-based architecture. Particle physics experiments are hitting a "hardware wall" as the industry shifts toward architectures with massive numbers of slower cores and specialized GPU accelerators. Traditional High-Energy Physics (HEP) software, often designed for serial execution, struggles to utilize these resources efficiently.

DUNE faces a specific "big data" challenge: its Liquid Argon Time Projection Chambers (LArTPCs) produce individual units of raw data that are gargantuan. A single 100-second readout from one 10-kiloton far detector module generates roughly 115 TB of data. Loading this into memory as a single "event" is impossible on standard hardware. To overcome this, the collaboration is re-architecting software to handle sub-event parallelism, manage shrinking RAM-per-core ratios, and balance workloads between CPUs and GPUs.

Methods: A Hybrid Software Strategy

The collaboration employs a hybrid software strategy centered on two main pillars: the art framework and the Wire-Cell Toolkit (WCT). While art manages the high-level event loop, WCT uses a data-flow graph model where "components" are nodes connected by edges. This allows for transverse parallelism—processing data from different sub-detectors, such as the 150 Anode Plane Assemblies (APAs) in DUNE, simultaneously.

Simplified WCT data-flow graph showing parallel processing of APA files.

Both art and WCT utilize Intel Threading Building Blocks (TBB) for multi-threading. This synergy allows them to share a common thread pool, avoiding the thread pool contention that occurs when different libraries compete for CPU cores. Additionally, portable GPU programming models like Kokkos and libtorch are used to port computationally intensive "hot spots," such as Fast Fourier Transforms (FFTs) and AI/ML inference, to accelerators.

Architectural Insights for HPC

A critical challenge in High-Performance Computing (HPC) is the "N-1" problem, where users request $N$ cores just to secure enough RAM for a single-threaded job, wasting $N-1$ cores. Multi-threaded execution is essential for "memory hygiene." In the WCT graph, data objects are immutable and handled via shared pointers, eliminating the need for complex mutexes that often degrade performance.

Distributed WCT graph architecture across heterogeneous hardware hosts.

As shown in the distributed architecture, the software can offload tasks to specialized hosts. By using TBB async_node, CPU cores can perform other work while waiting for GPU tasks to complete, maximizing total system utilization. This modularity allows the software to map the physical detector's structure directly into parallel software pipelines, preventing the "ruinous memory consumption" of monolithic data loading.

Key Results and Future Directions

Benchmarks show that GPU-based simulations run 18x faster than their CPU counterparts, while current GPU implementations for signal processing are roughly 2x faster. The graph-based architecture successfully maps DUNE’s physical detector modularity directly into software pipelines, providing a clear path to handling 115 TB data bursts.

Future work focuses on distributing the WCT graph across multiple compute nodes using frameworks like ADIOS2 and optimizing parallel I/O via HDF5 to prevent the data "sink" from becoming a bottleneck. Transitioning from simple semaphore-based GPU load balancing to a fully asynchronous model remains a primary objective for maximizing efficiency on next-generation supercomputers.

Summary updated

Abstract

DUNE, like other HEP experiments, faces a challenge related to matching execution patterns of our production simulation and data processing software to the limitations imposed by modern high-performance computing facilities. In order to efficiently exploit these new architectures, particularly those with high CPU core counts and GPU accelerators, our existing software execution models require adaptation. In addition, the large size of individual units of raw data from the far detector modules pose an additional challenge somewhat unique to DUNE. Here we describe some of these problems and how we begin to solve them today with existing software frameworks and toolkits. We also describe ways we may leverage these existing software architectures to attack remaining problems going forward. This whitepaper is a contribution to the Computational Frontier of Snowmass21.
BibTeX
@misc{fleming2022dune,
  author = {B. T. Fleming and K. Knoepfel and Meifeng Lin and X. Qian and Yihui Ren and Brett Viren and H. Wei and Shinjae Yoo and Haiwang Yu},
  title = {{DUNE Software and High Performance Computing}},
  year = {2022},
  doi = {10.48550/arxiv.2203.06104},
}
Figure