Abstract
EMPIRE is a state-of-the-art simulation code for electromagnetics and plasma physics, built to leverage modern supercomputing architectures. By utilizing the Kokkos library, EMPIRE provides performance portability across CPUs and GPUs, offering a unified framework for solving complex multi-physics problems. This manual details the code's capabilities, including advanced solver integration and the coupling of full-wave domains with 1D transmission lines and circuit networks.
Introduction
Simulating plasma physics and electromagnetics at scale is a massive computational challenge. Researchers often struggle with "performance portability"—the ability for a single code to run efficiently on different hardware, such as traditional Intel CPUs versus modern NVIDIA GPUs. Furthermore, many engineering problems require coupling different scales of physics, such as a large-scale electromagnetic field interacting with a specific circuit component or a transmission line.
EMPIRE addresses these gaps by providing a flexible, YAML-driven interface that handles everything from basic electrostatic solves to massive, parallelized electromagnetic simulations involving complex materials like isotropic cold plasma.
Methodology
EMPIRE is an ecosystem built on several high-performance libraries to ensure scalability and efficiency:
- Performance Portability: It uses the Kokkos library, allowing the code to run on diverse hardware architectures without significant rewrites.
- Discretization: The code supports both simplex (triangles/tetrahedrons) and tensor product (quadrilaterals/hexahedrons) meshes. It utilizes finite element methods, specifically supporting HCurl (edge) and HDiv (face) basis functions.
- Advanced Solvers: For linear systems, EMPIRE integrates with the Trilinos project, specifically using MueLu for Algebraic Multigrid (AMG) solvers. This is critical for scaling to millions of elements.
- Time Stepping: It offers sophisticated schemes, including the dissipative Friedman method (to smooth high-frequency noise), RK Implicit Midpoint, and SDIRK (Singly Diagonally Implicit Runge-Kutta) for L-stable performance.
- Coupling: A standout feature is the Circuit Network block, which allows users to model 1D transmission lines and Kirchhoff junctions that interact dynamically with the 3D electromagnetic domain.
Data Structures and Workflow
As a high-performance tool, EMPIRE utilizes structured input and output formats to manage complex simulation data:
- Input Formats: Simulations are defined using YAML files, chosen for their human readability and clean structure.
- Mesh Formats: It supports externally provided ExodusII files (
.gor.gen) and includes an internal Inline Mesh Generator for simpler geometries. - Output Formats: Results are typically written to Exodus files for field data and H5Part files for particle data.
- Python Integration: A robust Python utility module (
empire) automates the workflow of running simulations, archiving results, and performing statistical analysis on history files.
Key Insights and Performance
EMPIRE is designed with the future of supercomputing in mind. By building on Kokkos, the team ensures that the code remains relevant as hardware evolves from CPU-heavy to GPU-heavy architectures. The manual highlights "intelligent" defaults, such as using Edge-Based field pushes for electromagnetics but Nodal-Based pushes for electrostatics, reflecting the different mathematical requirements of each domain.
Furthermore, the inclusion of an "At Scale Multigrid" option specifically tuned for GPUs (via CUDA) demonstrates that the code is intended for the world's largest leadership-class computers. In terms of accuracy, users can expect second-order precision in the electric field when using uniform tensor product meshes and consistent second-order accuracy in voltage for electrostatic problems.
Limitations and Future Work
The EMPIRE suite is an evolving project with several areas identified for future development:
- High-Order Support: Support for high-order basis functions is currently limited; most electromagnetic simulations are restricted to first-order accuracy on simplex meshes.
- Mesh Constraints: EMPIRE does not currently support meshes with more than one type of element (e.g., mixing tetrahedrons and hexahedrons).
- Restart Compatibility: "Restart" files are not guaranteed to be backwards compatible with newer versions or cross-platform compatible between different filesystems.
- Dynamic Impedance: While the code supports impedance boundary conditions, it does not yet support dynamic impedance on those boundaries.
This mini paper is based on the EMPIRE User Manual from Sandia National Laboratories.
Summary updated