EMPIRE User Manual

Matthew Tyler Bettencourt, Keith Cartwright, Eric C. Cyr, N. Hamlin, Edward Love, W. Caleb McDaniel, Duncan McGregor, Scott T. Miller, Chris H. Moore, Roger P. Pawlowski, E. G. Phillips, T. D. Pointon, Gary A. Radke, Nicholas Roberds, Nathan V. Roberts, Steven J. Shields, Matthew Swan, C.d. Turner, David Sirajuddin, David Collins, David Stafford2022 · DOI

Sandia National Laboratories

Abstract

EMPIRE is a state-of-the-art simulation code for electromagnetics and plasma physics, built to leverage modern supercomputing architectures. By utilizing the Kokkos library, EMPIRE provides performance portability across CPUs and GPUs, offering a unified framework for solving complex multi-physics problems. This manual details the code's capabilities, including advanced solver integration and the coupling of full-wave domains with 1D transmission lines and circuit networks.

Introduction

Simulating plasma physics and electromagnetics at scale is a massive computational challenge. Researchers often struggle with "performance portability"—the ability for a single code to run efficiently on different hardware, such as traditional Intel CPUs versus modern NVIDIA GPUs. Furthermore, many engineering problems require coupling different scales of physics, such as a large-scale electromagnetic field interacting with a specific circuit component or a transmission line.

EMPIRE addresses these gaps by providing a flexible, YAML-driven interface that handles everything from basic electrostatic solves to massive, parallelized electromagnetic simulations involving complex materials like isotropic cold plasma.

Methodology

EMPIRE is an ecosystem built on several high-performance libraries to ensure scalability and efficiency:

  • Performance Portability: It uses the Kokkos library, allowing the code to run on diverse hardware architectures without significant rewrites.
  • Discretization: The code supports both simplex (triangles/tetrahedrons) and tensor product (quadrilaterals/hexahedrons) meshes. It utilizes finite element methods, specifically supporting HCurl (edge) and HDiv (face) basis functions.
  • Advanced Solvers: For linear systems, EMPIRE integrates with the Trilinos project, specifically using MueLu for Algebraic Multigrid (AMG) solvers. This is critical for scaling to millions of elements.
  • Time Stepping: It offers sophisticated schemes, including the dissipative Friedman method (to smooth high-frequency noise), RK Implicit Midpoint, and SDIRK (Singly Diagonally Implicit Runge-Kutta) for L-stable performance.
  • Coupling: A standout feature is the Circuit Network block, which allows users to model 1D transmission lines and Kirchhoff junctions that interact dynamically with the 3D electromagnetic domain.

Data Structures and Workflow

As a high-performance tool, EMPIRE utilizes structured input and output formats to manage complex simulation data:

  • Input Formats: Simulations are defined using YAML files, chosen for their human readability and clean structure.
  • Mesh Formats: It supports externally provided ExodusII files (.g or .gen) and includes an internal Inline Mesh Generator for simpler geometries.
  • Output Formats: Results are typically written to Exodus files for field data and H5Part files for particle data.
  • Python Integration: A robust Python utility module (empire) automates the workflow of running simulations, archiving results, and performing statistical analysis on history files.

Key Insights and Performance

EMPIRE is designed with the future of supercomputing in mind. By building on Kokkos, the team ensures that the code remains relevant as hardware evolves from CPU-heavy to GPU-heavy architectures. The manual highlights "intelligent" defaults, such as using Edge-Based field pushes for electromagnetics but Nodal-Based pushes for electrostatics, reflecting the different mathematical requirements of each domain.

Furthermore, the inclusion of an "At Scale Multigrid" option specifically tuned for GPUs (via CUDA) demonstrates that the code is intended for the world's largest leadership-class computers. In terms of accuracy, users can expect second-order precision in the electric field when using uniform tensor product meshes and consistent second-order accuracy in voltage for electrostatic problems.

Limitations and Future Work

The EMPIRE suite is an evolving project with several areas identified for future development:

  • High-Order Support: Support for high-order basis functions is currently limited; most electromagnetic simulations are restricted to first-order accuracy on simplex meshes.
  • Mesh Constraints: EMPIRE does not currently support meshes with more than one type of element (e.g., mixing tetrahedrons and hexahedrons).
  • Restart Compatibility: "Restart" files are not guaranteed to be backwards compatible with newer versions or cross-platform compatible between different filesystems.
  • Dynamic Impedance: While the code supports impedance boundary conditions, it does not yet support dynamic impedance on those boundaries.
Department of Energy and NNSA

This mini paper is based on the EMPIRE User Manual from Sandia National Laboratories.

Summary updated

Abstract

This is the user manual for EMPIRE, a simulation code for electromagnetics and plasma physics.
BibTeX
@misc{bettencourt2022empire,
  author = {Matthew Tyler Bettencourt and Keith Cartwright and Eric C. Cyr and N. Hamlin and Edward Love and W. Caleb McDaniel and Duncan McGregor and Scott T. Miller and Chris H. Moore and Roger P. Pawlowski and E. G. Phillips and T. D. Pointon and Gary A. Radke and Nicholas Roberds and Nathan V. Roberts and Steven J. Shields and Matthew Swan and C.D. Turner and David Sirajuddin and David Collins and David Stafford},
  title = {{EMPIRE User Manual}},
  year = {2022},
  doi = {10.2172/1900358},
}
Figure