TL;DR
This report assesses the current status of the EMPIRE (ElectroMagnetic Plasma In Realistic Environments) code, focusing on its particle-in-cell (PIC) components, linear solvers, and time integration capabilities. The findings highlight areas for improvement, particularly in linear solver performance, to meet future simulation demands for the ASC ATDM program.
What Problem Does This Paper Address?
The EMPIRE code is a critical tool for simulating plasma environments, and this report addresses the need to assess its readiness for upcoming milestones in the ASC ATDM program. Specifically, it evaluates the performance, verification status, and features of EMPIRE's electromagnetic and electrostatic PIC solutions, including its solvers, time integration, and checkpoint/restart mechanisms. The goal is to identify bottlenecks and guide future development to ensure EMPIRE can handle increasingly complex and large-scale simulations.
Methods
The paper details a comprehensive assessment of EMPIRE's capabilities. Key methods include:
- Performance Benchmarking: Evaluating the code's throughput (time-steps per second) on various hardware architectures (HSW and KNL) using realistic geometries ("blob" problem) and different mesh resolutions. This involves analyzing the performance of the PIC algorithm (particle move, sorting, weighting) and the linear solvers.
- Linear Solver Analysis: Investigating the performance of both electrostatic (ES) and electromagnetic (EM) linear systems, including the development and testing of new solver strategies and preconditioners.
- Time Integration: Integrating and testing the Tempus time-integration package within EMPIRE, evaluating various integrators (Leapfrog, IMEX-RK, HHT-α, Newmark-β, Operator-Split) for accuracy and performance on benchmark problems.
- Verification Studies: Employing regression, analytic, and convergence tests to ensure the code's accuracy and adherence to theoretical predictions for plasma physics phenomena like TEM waves and Langmuir waves.
- I/O Performance Evaluation: Assessing the efficiency of checkpointing and restart mechanisms, including the use of Cray DataWarp Burst Buffers and the FAODEL software library.
Datasets
The paper does not explicitly mention training datasets. For evaluation, it utilizes:
- "Blob" Problem: A realistic geometry used for performance benchmarking and scaling studies.
- Plasma Physics Test Cases:
- TEM Wave in Plasma: For verifying electromagnetic wave propagation.
- Langmuir Waves and Landau Damping: For assessing electrostatic wave behavior and kinetic effects.
- OscillatingEField1D Test: For basic electromagnetic behavior without particles.
These test cases are designed to represent a range of physics relevant to EMPIRE's applications and are compared against analytical solutions or established theoretical models.
Key Insights and Ideas
- Linear Solvers are the Bottleneck: For larger problem sizes, the linear solver time dominates the overall runtime, significantly exceeding particle update times. This is a critical area for future optimization.
- EMPIRE Scales Well (Mostly): The particle update component shows excellent weak and strong scaling to large numbers of nodes. However, linear solver scaling can be problematic, especially for larger meshes.
- Tempus Integration is Promising: The integration of the Tempus time-integration package provides EMPIRE with a wider array of advanced time-stepping methods, including higher-order and IMEX schemes, which are crucial for future development.
- I/O is a Significant Challenge: Checkpointing and restart operations, particularly particle serialization, present a substantial overhead. The use of Burst Buffers and FAODEL shows promise but requires further optimization.
- Verification Confirms Accuracy: The code demonstrates convergence to theoretical solutions for both electrostatic and electromagnetic problems, validating its accuracy for various physics regimes.
Key Results
- Performance: EMPIRE achieves 10s of time-steps per second on smaller problems, but scales down significantly for larger problems, with linear solver times being the primary contributor to this slowdown.
- Linear Solvers:
- ES solver improvements led to approximately 50% speedup for the largest XXL mesh.
- EM solver reformulation resulted in improved performance, though strong scaling still shows limitations for XL and XXL meshes.
- Time Integration: Tempus steppers were successfully integrated and demonstrated expected order of accuracy on benchmark tests.
- Verification: EMPIRE showed second-order spatial and time accuracy for electrostatic problems and second-order spatial and first-order time accuracy for electromagnetic problems, consistent with theoretical expectations.
- I/O: Burst Buffers provided up to a 4x improvement for H5Hut and 1.5x for FAODEL. FAODEL was found to be significantly faster than H5Hut for checkpointing.
Limitations and Future Work
- Linear Solver Performance: The report explicitly states that the performance of the linear solver will need to be a major focus in FY19 to meet future milestones.
- EM Solver Scaling: While improved, the EM solver still faces scaling challenges for larger meshes.
- I/O Limitations:
- Restart Capability: Current checkpoint/restart infrastructure needs revisions to better support electromagnetic simulations and a more robust field data interface.
- RNG Storage: The inability to checkpoint random number generator states means restarts are not reproducible. This needs to be implemented in FY19.
- Per-Node Particle Size: A limitation on object size in FAODEL restricts the number of particles per rank, creating memory pressure. This is deferred to FY19.
- Serialization Performance: The current serial implementation of particle serialization is a bottleneck, especially on KNL. Parallelization or leveraging native Kokkos Views is planned.
- Scale Testing: While small-scale I/O tests were performed, larger-scale testing is needed to confirm performance at scale.
- Job-to-Job Handoffs: A known bug on Cray platforms prevents job-to-job communication, hindering the use of FAODEL for migrating I/O and analysis tasks. This will be revisited after a Cray update.
- Future Work:
- Integrate Tempus into EMPIRE-Fluids and EMPIRE-Hybrid codes.
- Develop and implement a second-order electromagnetic time integrator.
- Continue to optimize linear solvers and I/O performance.
- Address the identified limitations in checkpoint/restart and serialization.
Summary updated