LAMMPS/Kokkos/HIPPYNN Sierra Timings [Slides]

Benjamin Nebgen2023 · DOI

Abstract

This report evaluates the performance of the HIPPYNN machine learning interatomic potential (MLIAP) within the LAMMPS molecular dynamics framework, utilizing Kokkos for GPU acceleration on the Sierra supercomputer. While the computational throughput scales impressively across thousands of GPUs, the study reveals a critical bottleneck: parallel I/O performance fails to keep pace with the simulation speed, posing a significant challenge for billion-atom simulations.

Introduction

Molecular Dynamics (MD) simulations are essential for understanding material behavior at the atomic level. Traditionally, researchers have faced a trade-off between the speed of empirical potentials and the accuracy of quantum mechanical methods. Machine Learning Interatomic Potentials (MLIAPs), such as HIPPYNN, bridge this gap by offering near-quantum accuracy at a significantly lower computational cost. However, as the field moves toward "exascale" simulations involving billions of atoms, the underlying software stack must demonstrate efficient scaling across massive supercomputing architectures. This study investigates the performance and scalability of the LAMMPS/Kokkos/HIPPYNN stack on the Sierra supercomputer.

Methods

The evaluation utilized a high-performance computing stack optimized for GPU-accelerated science:

  • LAMMPS: The industry-standard molecular dynamics simulator.
  • Kokkos: A programming model and performance portability layer that allows the code to run efficiently on the NVIDIA V100 GPUs within Sierra nodes.
  • HIPPYNN: A deep learning architecture designed for atomistic outputs.

The researchers conducted weak scaling tests on an Ionic Liquid system. In these tests, the total system size was increased proportionally with the number of nodes (up to 2,048 Sierra nodes), maintaining a constant workload per GPU. The scale ranged from a single GPU to approximately 8,192 GPUs (4 GPUs per node), culminating in a massive 1-billion atom simulation.

Key Results and Analysis

Computational Throughput and Scaling

The core molecular dynamics calculations demonstrated excellent weak scaling. The system achieved a peak throughput of over $10^{10}$ atom-steps per second at the highest GPU counts. Parallel efficiency remained remarkably high (near 1.0) for up to approximately 1,000 GPUs. While a gradual decline in efficiency was observed beyond this point, the scaling remained functional even at the $10^4$ GPU limit. These results indicate that the HIPPYNN/Kokkos implementation is highly optimized for the compute-intensive portions of the simulation.

The I/O Bottleneck

Despite the impressive compute performance, the study identified a significant bottleneck in data management. The MPIIO (Message Passing Interface I/O) package, intended for high-performance parallel writing, did not perform as expected. Writing a single trajectory frame for the 1-billion atom system (approximately 40GB) required 4 minutes. Crucially, write times scaled linearly with system size rather than remaining flat, suggesting that the I/O process was effectively serialized or severely bottlenecked. This limitation ultimately led to the failure of the 2,048-node run during the output file writing phase.

Initialization Overhead

At the largest scales, the time required to instantiate the MLIAP network across thousands of nodes became a non-trivial factor. For the 2,048-node test, timing included the first 750 steps to account for this setup. This initialization overhead resulted in a noticeable dip in "apparent" performance, highlighting that setup time can become a significant fraction of total runtime in extreme-scale environments.

Conclusion and Future Work

The combination of LAMMPS, Kokkos, and HIPPYNN is capable of reaching the billion-atom scale with high computational efficiency on the Sierra supercomputer. However, the transition to stable exascale MD is currently hindered by parallel I/O limitations and initialization overhead.

Future work will focus on:

  1. Optimizing MPIIO: Investigating why the parallel I/O package is scaling linearly rather than providing the expected parallel speedup.
  2. Initialization Efficiency: Reducing the time required to instantiate the MLIAP network across thousands of nodes.
  3. Fault Tolerance: Addressing the stability of the simulation when attempting to write massive datasets to disk at the scale of 2,048 nodes.

Summary updated

Abstract

Abstract not provided.
BibTeX
@misc{nebgen2023lammps,
  author = {Benjamin Nebgen},
  title = {{LAMMPS/Kokkos/HIPPYNN Sierra Timings [Slides]}},
  year = {2023},
  doi = {10.2172/1988544},
}
Figure