Abstract
Cosmological N-body simulations are essential for understanding the Large-Scale Structure (LSS) of the universe. However, the memory requirements of traditional algorithms often act as a "memory wall," limiting the scale of simulations on current supercomputing architectures. We present CUBE, a two-level particle-mesh (PM) algorithm designed to minimize memory footprint and maximize scaling efficiency. By employing fixed-point compression and hardware-level optimizations, CUBE reduces memory usage to just 6 bytes per particle. We demonstrate its capabilities through Cosmo-π, a 4.4-trillion-particle simulation that achieves 95% weak-scaling efficiency on 20,480 cores.
Introduction
To achieve the precision required for next-generation galaxy surveys, astrophysicists must track the evolution of trillions of particles ($N > 10^{12}$) over cosmic time. While raw computational power continues to grow, memory capacity remains a primary bottleneck. Traditional N-body codes store particle data in high-precision floating-point formats, typically requiring 24 to 48 bytes per particle (bpp). This high memory demand often leads to underutilization of CPU cycles as systems run out of RAM. CUBE addresses this challenge by optimizing information storage and leveraging modern vector instructions to break the "memory wall" without sacrificing scientific accuracy.
Methodology
CUBE introduces several innovations to optimize both memory and performance:
Two-Level Particle-Mesh (PMPM)
CUBE decomposes gravitational forces into long-range and short-range components. Long-range forces are computed on a coarse global mesh, while short-range forces are calculated locally on fine-mesh "tiles." This localization significantly reduces the need for global communication and allows for better cache utilization, enhancing overall parallel efficiency.
Fixed-Point Compression
The core innovation of CUBE is its information-optimized storage. Instead of absolute global coordinates, CUBE stores particle positions as relative offsets from a "parent cell" in the coarse mesh. By using 1-byte integers for these offsets, the phase-space memory footprint is reduced to 6 bpp—an order of magnitude improvement over traditional methods. This approach recognizes that cosmological simulations require correct statistical distributions rather than perfect individual trajectories.
Hardware-Level Optimization
CUBE repurposes Intel AVX512 VNNI (Vector Neural Network Instructions) to accelerate mixed-precision distance calculations. Originally intended for AI workloads, these instructions allow CUBE to achieve a 3x throughput increase over standard scalar instructions. Furthermore, computationally expensive math functions are replaced with fast, domain-specific approximations that preserve scientific accuracy while saving CPU cycles.
The Cosmo-π Simulation
The efficacy of CUBE is demonstrated through the Cosmo-π simulation, which tracks 4.4 trillion cold dark matter (CDM) particles within a cubic volume of 3,200 Mpc/h. This simulation evolves the universe from a redshift of $z=99$ to the present day ($z=0$).
Cosmo-π represents a significant leap in scale, surpassing previous benchmarks such as TianNu (2.97 trillion particles) and Outer Rim (1.07 trillion particles). The simulation provides a high-resolution view of the cosmic web, including nodes, filaments, and voids.
Results and Performance
CUBE achieves a total memory consumption of 12.8 bpp, including all buffers and mesh overhead. This is approximately 14.5 times more efficient than the TianNu simulation (186 bpp).
Scalability
The algorithm exhibits near-perfect linear weak scaling. On the π 2.0 supercomputer, CUBE maintained 95% efficiency when scaling to 20,480 cores (512 nodes). This performance is facilitated by a 3D MPI rank distribution that ensures network proximity for processes managing adjacent physical volumes.
Scientific Validation
The scientific integrity of the compressed data was verified by comparing the resulting power spectrum $\Delta^2(k)$ against linear and non-linear theoretical predictions. The results confirm that CUBE's compression techniques do not degrade the statistical accuracy required for cosmological research, matching theoretical expectations across a wide range of scales.
Conclusion
CUBE demonstrates that by optimizing for memory and leveraging specialized hardware instructions, it is possible to run trillion-particle simulations with high efficiency on existing supercomputers. While the current force resolution is optimized for large-scale structures, future iterations of CUBE are poised for exascale systems, aiming for the 10-trillion-particle milestone necessary for upcoming "Stage IV" cosmological surveys.
Summary updated