Introduction
In high-performance computing (HPC), the lack of a universal job scheduling standard presents a significant barrier to software portability. Different supercomputing sites utilize various Local Resource Managers (LRMs) such as Slurm, PBS, LSF, or Cobalt. For a scientific application to be truly portable—capable of running seamlessly from a local workstation to a leadership-class supercomputer—it must implement a Job Abstraction API (JAAPI) to translate abstract job requests into machine-specific commands.
Historically, major workflow projects like Parsl, Swift/T, and RADICAL-Pilot have developed their own custom JAAPIs. This fragmented landscape leads to "costly redundancy," where developers spend excessive time maintaining boilerplate code for diverse schedulers rather than focusing on core research. Previous attempts at standardization, such as DRMAA or SAGA, have often struggled with complexity or sustainability issues.
System Architecture and Design
To address these challenges, the authors propose PSI/J (Portable Submission Interface for Jobs), a minimalistic, language-agnostic API designed to standardize job management. PSI/J is built on three core design principles:
- Minimalism: It serves as the "thinnest" possible layer, focusing strictly on job submission, monitoring, and state management.
- Layered Architecture: The system is organized into Local, Remote, and Nested layers. The Local layer interacts directly with schedulers on the same machine, while the Remote layer handles submissions via protocols like SSH.
- Decoupled Execution: PSI/J separates the Executor (which communicates with the scheduler) from the Launcher (which initiates multi-node processes like
mpirun).
The API is designed for asynchrony from the ground up, allowing it to handle thousands of concurrent jobs without blocking the main application. Furthermore, it mandates support for bulk status querying to prevent overwhelming schedulers with individual requests.
Evaluation and Real-World Integration
PSI/J was evaluated on several Oak Ridge National Laboratory (ORNL) supercomputers, including Crusher, Frontier, and Summit. The system's performance was validated through integration into production-grade frameworks such as Parsl, RADICAL-Pilot, Swift/T, and OSPREY.
Key performance results include:
- Minimal Overhead: The Python implementation adds only ~10 ms per job submission.
- Launcher Efficiency: Specialized launcher scripts introduce a negligible 2 ms overhead compared to raw subprocess calls.
- Scalability: Experiments showed that managing up to 100 concurrent jobs did not cause observable latency spikes in native system commands like
qstat.
Discussion and Future Directions
The "JAAPI Condition" identified by the authors suggests that the failure of previous standards was often structural rather than technical; high complexity leads to unsustainable maintenance costs. PSI/J’s minimalism is a strategic choice for long-term viability in the research software ecosystem.
While the Local layer is mature, future work focuses on finalizing the Remote and Nested layers. Additionally, while a reference Python implementation exists, the community aims to develop native bindings for other languages like C++ and Java to ensure broader adoption across the HPC landscape.
Summary updated