PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs

Mihael Hategan-Marandiuc, André Merzky, Nicholson T. Collier, Ketan Maheshwari, Jonathan Ozik, Matteo Turilli, Andreas Wilke, Justin M. Wozniak, Kyle Chard, Ian T. Foster, Rafael Ferreira da Silva, Shantenu Jha, Daniel E. Laney

Abstract

It is generally desirable for high-performance computing (HPC) applications to be portable between HPC systems, for example to make use of more performant hardware, make effective use of allocations, and to co-locate compute jobs with large datasets. Unfortunately, moving scientific applications between HPC systems is challenging for various reasons, most notably that HPC systems have different HPC schedulers. We introduce PSI/J, a job management abstraction API intended to simplify the construction of software components and applications that are portable over various HPC scheduler implementations. We argue that such a system is both necessary and that no viable alternative currently exists. We analyze similar notable APIs and attempt to determine the factors that influenced their evolution and adoption by the HPC community. We base the design of PSI/J on that analysis. We describe how PSI/J has been integrated in three workflow systems and one application, and also show via experiments that PSI/J imposes minimal overhead.
BibTeX
@inproceedings{hateganmarandiuc2023psi,
  author = {Mihael Hategan-Marandiuc and André Merzky and Nicholson T. Collier and Ketan Maheshwari and Jonathan Ozik and Matteo Turilli and Andreas Wilke and Justin M. Wozniak and Kyle Chard and Ian T. Foster and Rafael Ferreira da Silva and Shantenu Jha and Daniel E. Laney},
  title = {{PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs}},
  booktitle = {e-Science},
  year = {2023},
  doi = {10.1109/e-science58273.2023.10254912},
}
Figure