CUBE - Towards an Optimal Scaling of Cosmological N-body Simulations

Shenggan Cheng, Hao-Ran Yu, Derek Inman, Qiucheng Liao, Qiaoya Wu, James Lin

Abstract

Cosmological N-body simulations are essential for understanding the Large-Scale Structure (LSS) of the universe. However, the memory requirements of traditional algorithms often act as a "memory wall," limiting the scale of simulations on current supercomputing architectures. We present CUBE, a two-level particle-mesh (PM) algorithm designed to minimize memory footprint and maximize scaling efficiency. By employing fixed-point compression and hardware-level optimizations, CUBE reduces memory usage to just 6 bytes per particle. We demonstrate its capabilities through Cosmo-π, a 4.4-trillion-particle simulation that achieves 95% weak-scaling efficiency on 20,480 cores.

Introduction

To achieve the precision required for next-generation galaxy surveys, astrophysicists must track the evolution of trillions of particles ($N > 10^{12}$) over cosmic time. While raw computational power continues to grow, memory capacity remains a primary bottleneck. Traditional N-body codes store particle data in high-precision floating-point formats, typically requiring 24 to 48 bytes per particle (bpp). This high memory demand often leads to underutilization of CPU cycles as systems run out of RAM. CUBE addresses this challenge by optimizing information storage and leveraging modern vector instructions to break the "memory wall" without sacrificing scientific accuracy.

Methodology

CUBE introduces several innovations to optimize both memory and performance:

Two-Level Particle-Mesh (PMPM)

CUBE decomposes gravitational forces into long-range and short-range components. Long-range forces are computed on a coarse global mesh, while short-range forces are calculated locally on fine-mesh "tiles." This localization significantly reduces the need for global communication and allows for better cache utilization, enhancing overall parallel efficiency.

Fixed-Point Compression

The core innovation of CUBE is its information-optimized storage. Instead of absolute global coordinates, CUBE stores particle positions as relative offsets from a "parent cell" in the coarse mesh. By using 1-byte integers for these offsets, the phase-space memory footprint is reduced to 6 bpp—an order of magnitude improvement over traditional methods. This approach recognizes that cosmological simulations require correct statistical distributions rather than perfect individual trajectories.

Hardware-Level Optimization

CUBE repurposes Intel AVX512 VNNI (Vector Neural Network Instructions) to accelerate mixed-precision distance calculations. Originally intended for AI workloads, these instructions allow CUBE to achieve a 3x throughput increase over standard scalar instructions. Furthermore, computationally expensive math functions are replaced with fast, domain-specific approximations that preserve scientific accuracy while saving CPU cycles.

The Cosmo-π Simulation

The efficacy of CUBE is demonstrated through the Cosmo-π simulation, which tracks 4.4 trillion cold dark matter (CDM) particles within a cubic volume of 3,200 Mpc/h. This simulation evolves the universe from a redshift of $z=99$ to the present day ($z=0$).

Visual representation of the Cosmo-π simulation showing the cosmic web at various scales, from 400 Mpc/h down to 5 Mpc/h.

Cosmo-π represents a significant leap in scale, surpassing previous benchmarks such as TianNu (2.97 trillion particles) and Outer Rim (1.07 trillion particles). The simulation provides a high-resolution view of the cosmic web, including nodes, filaments, and voids.

Results and Performance

CUBE achieves a total memory consumption of 12.8 bpp, including all buffers and mesh overhead. This is approximately 14.5 times more efficient than the TianNu simulation (186 bpp).

Scalability

The algorithm exhibits near-perfect linear weak scaling. On the π 2.0 supercomputer, CUBE maintained 95% efficiency when scaling to 20,480 cores (512 nodes). This performance is facilitated by a 3D MPI rank distribution that ensures network proximity for processes managing adjacent physical volumes.

Weak and strong scaling performance of the CUBE algorithm. The top panel shows near-ideal weak scaling efficiency (95%) up to 20,480 cores.

Scientific Validation

The scientific integrity of the compressed data was verified by comparing the resulting power spectrum $\Delta^2(k)$ against linear and non-linear theoretical predictions. The results confirm that CUBE's compression techniques do not degrade the statistical accuracy required for cosmological research, matching theoretical expectations across a wide range of scales.

Conclusion

CUBE demonstrates that by optimizing for memory and leveraging specialized hardware instructions, it is possible to run trillion-particle simulations with high efficiency on existing supercomputers. While the current force resolution is optimized for large-scale structures, future iterations of CUBE are poised for exascale systems, aiming for the 10-trillion-particle milestone necessary for upcoming "Stage IV" cosmological surveys.

Summary updated

Abstract

N-body simulations are essential tools in physical cosmology to understand the large-scale structure (LSS) formation of the universe. Large-scale simulations with high resolution are important for exploring the substructure of universe and for determining fundamental physical parameters like neutrino mass. However, traditional particle-mesh (PM) based algorithms use considerable amounts of memory, which limits the scalability of simulations. Therefore, we designed a two-level PM algorithm CUBE towards optimal performance in memory consumption reduction. By using the fixed-point compression technique, CUBE reduces the memory consumption per N-body particle to only 6 bytes, an order of magnitude lower than the traditional PM-based algorithms. We scaled CUBE to 512 nodes (20,480 cores) on an Intel Cascade Lake based supercomputer with ≃95% weak-scaling efficiency. This scaling test was performed in Cosmo-π - a cosmological LSS simulation using ≃4.4 trillion particles, tracing the evolution of the universe over ≃13.7 billion years. To our best knowledge, Cosmo-π is the largest completed cosmological N-body simulation. We believe CUBE has a huge potential to scale on exascale supercomputers for larger simulations.
BibTeX
@inproceedings{cheng2020cube,
  author = {Shenggan Cheng and Hao-Ran Yu and Derek Inman and Qiucheng Liao and Qiaoya Wu and James Lin},
  title = {{CUBE - Towards an Optimal Scaling of Cosmological N-body Simulations}},
  booktitle = {CCGRID},
  year = {2020},
  url = {https://openreview.net/forum?id=P-PDaHlhd9},
}
Figure