A Performance Analysis of Task Scheduling for UQ Workflows on HPC Systems

Chung Ming Loi, Anne Reinarz, Mikkel Lykkegaard, W. A. Hornsby, John L. Buchanan, Linus SeelingerarXiv 2025 · DOI

TL;DR

This paper introduces a novel load balancing approach for Uncertainty Quantification (UQ) workflows on High-Performance Computing (HPC) systems, specifically integrating the UM-Bridge framework with HyperQueue (HQ) and SLURM. The proposed method significantly reduces scheduling overhead and improves overall execution time for UQ tasks, demonstrating a substantial improvement over traditional SLURM scheduling.

What Problem Does This Paper Address?

Modern scientific and engineering research increasingly relies on Uncertainty Quantification (UQ) to understand the reliability of models and predictions. UQ often involves running simulations thousands or even millions of times with varying input parameters. These simulations can be computationally demanding, and their runtimes can vary unpredictably. Traditional HPC schedulers like SLURM are designed for large, homogeneous jobs and struggle to efficiently manage these large numbers of small, potentially heterogeneous tasks. Users often resort to manual job submission or aggregation, leading to suboptimal resource utilization and increased complexity. This paper addresses the challenge of efficiently scheduling these UQ workloads on HPC systems without requiring system-level changes or extensive user expertise.

Methods

The authors propose a load balancing approach that works on top of existing HPC schedulers, specifically SLURM. They leverage the UM-Bridge framework, which provides a language-agnostic interface for UQ algorithms and simulation models via an HTTP protocol. The core of their solution is the integration of HyperQueue (HQ), a lightweight scheduler with minimal dependencies, which acts as a load balancer.

Here's how it works:

  • UM-Bridge: Acts as a universal interface, allowing UQ software and simulation models to communicate.
  • HyperQueue (HQ): Manages task scheduling and distribution. It can dynamically spawn model server instances on HPC nodes by submitting jobs to SLURM. HQ offers more flexibility than SLURM's job arrays, particularly in specifying job time requests and limits, which helps in efficiently handling tasks with variable runtimes.
  • SLURM Integration: The HQ-based load balancer submits jobs to SLURM for resource allocation. This approach avoids the significant overhead associated with SLURM submitting individual jobs for each evaluation.

The paper also discusses containerization (using Singularity) for reproducibility and system independence, which is compatible with both Kubernetes and HPC environments.

Datasets

The paper does not explicitly mention training datasets. The evaluation is performed using two primary benchmarks:

  • GS2 Gyrokinetic Plasma Turbulence Simulator: A computationally intensive simulation with highly variable runtimes (minutes to hours) depending on input parameters.
  • Gaussian Process (GP) Surrogate of GS2: A computationally inexpensive model derived from GS2, used to approximate its behavior.

These benchmarks are used to simulate UQ workloads, where the GS2 simulator or its GP surrogate are evaluated multiple times with varying input parameters. The authors also mention a simpler benchmark involving solving eigenproblems for randomly generated square matrices (eigen-100 and eigen-5000) to represent less computationally intensive tasks.

Key Insights and Ideas

  • The "Small Task" Problem: Traditional HPC schedulers are ill-suited for the growing prevalence of UQ workloads, which consist of many small, often unpredictable tasks.
  • Overhead is Key: For UQ tasks, the scheduling overhead can be a significant bottleneck, often outweighing the actual computation time, especially for faster jobs.
  • HQ's Advantage: HyperQueue's ability to request a single, larger SLURM allocation and then manage tasks within that allocation dramatically reduces per-job overhead compared to SLURM submitting individual jobs.
  • Flexible Job Timing: HQ's ability to specify both a job time request (expected duration) and a job time limit (maximum duration) is crucial for efficiently handling tasks with variable runtimes, preventing resource underutilization.
  • UM-Bridge as an Enabler: UM-Bridge provides the necessary abstraction to decouple UQ algorithms from simulation models, making it easier to integrate with load balancing solutions like HQ.

Key Results

The paper demonstrates significant improvements in scheduling efficiency for UQ workflows:

  • Scheduling Overhead Reduction: The HQ-based load balancer reduced scheduling overhead by up to three orders of magnitude compared to a naive SLURM approach.
  • Makespan Reduction: For long-running simulations like GS2, the mean makespan (total execution time) was reduced by up to 38% for both 2 and 10 jobs filling the queue.
  • Faster Jobs: For very fast jobs (e.g., eigen-100), the HQ approach was up to three times quicker in terms of makespan, despite a slight increase in CPU time due to model initiation overhead.
  • Schedule Length Ratio (SLR): The SLR metric, which accounts for scheduler overhead, showed that HQ consistently outperformed SLURM, indicating better overall efficiency.

Limitations and Future Work

  • Model Initiation Overhead: The current HQ implementation incurs a ~1-second overhead for initializing model servers for each job. This is noticeable for very fast jobs and can be mitigated by implementing persistent servers.
  • Filesystem Dependencies: The authors encountered issues with filesystem updates for communication between the load balancer and model servers, requiring manual workarounds. They are working on a more robust network-based solution.
  • Complex Workflows: The current framework is primarily tested on embarrassingly parallel tasks. Future work will focus on handling more complex workflows with interdependencies and dynamic scheduling requirements, such as adaptive GP models.
  • Benchmark Diversity: The authors acknowledge that their chosen benchmarks might not be representative of all UQ workloads and plan to incorporate a more diverse mixture in future work.

Summary updated

Abstract

Uncertainty Quantification (UQ) workloads are becoming increasingly common in science and engineering. They involve the submission of thousands or even millions of similar tasks with potentially unpredictable runtimes, where the total number is usually not known a priori. A static one-size-fits-all batch script would likely lead to suboptimal scheduling, and native schedulers installed on High Performance Computing (HPC) systems such as SLURM often struggle to efficiently handle such workloads. In this paper, we introduce a new load balancing approach suitable for UQ workflows. To demonstrate its efficiency in a real-world setting, we focus on the GS2 gyrokinetic plasma turbulence simulator. Individual simulations can be computationally demanding, with runtimes varying significantly-from minutes to hours-depending on the high-dimensional input parameters. Our approach uses UQ and Modelling Bridge, which offers a language-agnostic interface to a simulation model, combined with HyperQueue which works alongside the native scheduler. In particular, deploying this framework on HPC systems does not require system-level changes. We benchmark our proposed framework against a standalone SLURM approach using GS2 and a Gaussian Process surrogate thereof. Our results demonstrate a reduction in scheduling overhead by up to three orders of magnitude and a maximum reduction of 38% in CPU time for long-running simulations compared to the naive SLURM approach, while making no assumptions about the job submission patterns inherent to UQ workflows.
BibTeX
@misc{loi2025performance,
  author = {Chung Ming Loi and Anne Reinarz and Mikkel Lykkegaard and W. A. Hornsby and John L. Buchanan and Linus Seelinger},
  title = {{A Performance Analysis of Task Scheduling for UQ Workflows on HPC Systems}},
  year = {2025},
  eprint = {2503.22645},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.22645},
  howpublished = {arXiv},
}
Figure