Large Scale ARM Computing Cluster and its Application in HEP

Yaosong Cheng, Wei Sun, Yujiang Bi, Yaodong Cheng, Jingyan Shi, Lu Wang, Qiuling Yao, Qingbao Hu, Minxing Zhang2022 · DOI

Abstract

With the advancement of several large scientific experiments in the field of high-energy physics, the speed of data generation continues to increase and will reach exabyte (EB) levels in the next decade. This puts forward higher requests for data storage and computation processing. Reducing reliance on a single chip architecture can provide more cost-effective storage and computing solutions. We built a supercomputer cluster with 9,600 computing cores based on the ARM architecture in Dongguan, China. On this ARM cluster, we ported some common software for high-energy physics and made some optimizations for Lattice quantum chromodynamics (LQCD). In addition, we compared the performance of ARM and X86 in many application scenarios, which is beneficial to improve computational efficiency.

Introduction

High-Energy Physics (HEP) is entering the era of "Exabyte Computing." Modern experiments such as the Large High Altitude Air Shower Observatory (LHAASO) and the Jiangmen Underground Neutrino Observatory (JUNO) generate staggering amounts of data that require immense storage and processing power. For decades, the scientific community has relied almost exclusively on the x86 (Intel/AMD) architecture. However, as power costs rise and the need for cost-effective scaling grows, this monoculture has become a bottleneck. This research explores whether ARM architecture—traditionally used in mobile devices but now maturing into server-grade hardware—can provide a viable, high-performance alternative for the specialized software stacks used in HEP.

System Architecture and Software Porting

The researchers deployed a large-scale cluster in Dongguan, China, consisting of 100 compute nodes powered by Huawei Kunpeng 920 processors (ARMv8.2). Each node features two 48-core CPUs (96 cores total per node) and 256 GB of RAM, interconnected via a 100Gb RoCE (RDMA over Converged Ethernet) network.

The software and hardware ecosystem deployed on the ARM cluster, including ported HEP applications and system software.

A critical component of this work involved porting the HEP software ecosystem from x86 to the aarch64 architecture. This included the ROOT data analysis framework, the GEANT simulation toolkit, and the EOS storage system. Porting was not merely a matter of recompilation; for instance, the EOS system required manual rewriting of Intel-specific assembly code (specifically for CRC32 functions) into ARM instructions to maintain data integrity and performance. Furthermore, Lattice Quantum Chromodynamics (LQCD) applications were optimized by integrating the Kunpeng Math Library (KML) and fine-tuning MPI communication patterns, resulting in a 7% performance boost.

Performance Evaluation

The cluster's performance was evaluated using standard HEP benchmarks (HEP-SPEC06), real-world LHAASO-WCDA reconstruction tasks, and memory bandwidth tests (STREAM).

Throughput vs. Single-Core Speed

A primary finding of the study is the trade-off between single-core speed and total node throughput. While an individual ARM core is approximately 40% slower than a high-end Intel core for serialized tasks, the ARM server's higher core density allows it to excel in parallel workloads. In the HEP-SPEC06 test, the 128-core ARM server outperformed an Intel Gold 6230 server by 2.02x in total throughput.

Comparison of reconstruction time for a single job, illustrating the performance gap in single-core execution between ARM and x86.

Memory and Scaling Advantages

The Kunpeng 920 processors demonstrated over 20% higher memory bandwidth than comparable Intel Xeon Gold processors. This is particularly beneficial for data-intensive scientific computing where data movement is often the primary bottleneck. Additionally, the ARM architecture showed superior "weak and strong scaling" in Chroma tests, maintaining efficiency more effectively than x86 as problem sizes or node counts increased.

Conclusion

The deployment of this 9,600-core ARM cluster demonstrates that ARM architecture is a viable and cost-effective alternative for high-throughput HEP workloads. While single-core performance remains a limitation for non-parallelizable tasks, the advantages in core density, memory bandwidth, and scaling efficiency make it a strong candidate for future exabyte-scale scientific computing. Future work will focus on expanding multi-node parallel applications and further stabilizing the ARM ecosystem to support a broader range of experiments.

Summary updated

Abstract

With the advancement of several large scientific experiments in the field of high-energy physics, the speed of data generation continues to increase and will reach exabyte(EB) levels in the next decade. This puts forward higher requests for data storage and computation processing. Reducing reliance on a single chip architecture can provide more cost-effective storage and computing solutions. We built a supercomputer cluster with 9,600 computing cores based on the ARM architecture in Dongguan, China. On this ARM cluster, we ported some common software for high-energy physics and made some optimizations for Lattice quantum chromodynamics(LQCD). In addition, we compared the performance of ARM and X86 in many application scenarios, which is beneficial to improve computational efficiency.
BibTeX
@misc{cheng2022large,
  author = {Yaosong Cheng and Wei Sun and Yujiang Bi and Yaodong Cheng and Jingyan Shi and Lu Wang and Qiuling Yao and Qingbao Hu and Minxing Zhang},
  title = {{Large Scale ARM Computing Cluster and its Application in HEP}},
  year = {2022},
  doi = {10.22323/1.415.0015},
}
Figure