TL;DR
This paper introduces a novel load balancing approach for Uncertainty Quantification (UQ) workflows on High-Performance Computing (HPC) systems, specifically integrating the UM-Bridge framework with HyperQueue (HQ) and SLURM. The proposed method significantly reduces scheduling overhead and improves overall execution time for UQ tasks, demonstrating a substantial improvement over traditional SLURM scheduling.
What Problem Does This Paper Address?
Modern scientific and engineering research increasingly relies on Uncertainty Quantification (UQ) to understand the reliability of models and predictions. UQ often involves running simulations thousands or even millions of times with varying input parameters. These simulations can be computationally demanding, and their runtimes can vary unpredictably. Traditional HPC schedulers like SLURM are designed for large, homogeneous jobs and struggle to efficiently manage these large numbers of small, potentially heterogeneous tasks. Users often resort to manual job submission or aggregation, leading to suboptimal resource utilization and increased complexity. This paper addresses the challenge of efficiently scheduling these UQ workloads on HPC systems without requiring system-level changes or extensive user expertise.
Methods
The authors propose a load balancing approach that works on top of existing HPC schedulers, specifically SLURM. They leverage the UM-Bridge framework, which provides a language-agnostic interface for UQ algorithms and simulation models via an HTTP protocol. The core of their solution is the integration of HyperQueue (HQ), a lightweight scheduler with minimal dependencies, which acts as a load balancer.
Here's how it works:
- UM-Bridge: Acts as a universal interface, allowing UQ software and simulation models to communicate.
- HyperQueue (HQ): Manages task scheduling and distribution. It can dynamically spawn model server instances on HPC nodes by submitting jobs to SLURM. HQ offers more flexibility than SLURM's job arrays, particularly in specifying job time requests and limits, which helps in efficiently handling tasks with variable runtimes.
- SLURM Integration: The HQ-based load balancer submits jobs to SLURM for resource allocation. This approach avoids the significant overhead associated with SLURM submitting individual jobs for each evaluation.
The paper also discusses containerization (using Singularity) for reproducibility and system independence, which is compatible with both Kubernetes and HPC environments.
Datasets
The paper does not explicitly mention training datasets. The evaluation is performed using two primary benchmarks:
- GS2 Gyrokinetic Plasma Turbulence Simulator: A computationally intensive simulation with highly variable runtimes (minutes to hours) depending on input parameters.
- Gaussian Process (GP) Surrogate of GS2: A computationally inexpensive model derived from GS2, used to approximate its behavior.
These benchmarks are used to simulate UQ workloads, where the GS2 simulator or its GP surrogate are evaluated multiple times with varying input parameters. The authors also mention a simpler benchmark involving solving eigenproblems for randomly generated square matrices (eigen-100 and eigen-5000) to represent less computationally intensive tasks.
Key Insights and Ideas
- The "Small Task" Problem: Traditional HPC schedulers are ill-suited for the growing prevalence of UQ workloads, which consist of many small, often unpredictable tasks.
- Overhead is Key: For UQ tasks, the scheduling overhead can be a significant bottleneck, often outweighing the actual computation time, especially for faster jobs.
- HQ's Advantage: HyperQueue's ability to request a single, larger SLURM allocation and then manage tasks within that allocation dramatically reduces per-job overhead compared to SLURM submitting individual jobs.
- Flexible Job Timing: HQ's ability to specify both a job time request (expected duration) and a job time limit (maximum duration) is crucial for efficiently handling tasks with variable runtimes, preventing resource underutilization.
- UM-Bridge as an Enabler: UM-Bridge provides the necessary abstraction to decouple UQ algorithms from simulation models, making it easier to integrate with load balancing solutions like HQ.
Key Results
The paper demonstrates significant improvements in scheduling efficiency for UQ workflows:
- Scheduling Overhead Reduction: The HQ-based load balancer reduced scheduling overhead by up to three orders of magnitude compared to a naive SLURM approach.
- Makespan Reduction: For long-running simulations like GS2, the mean makespan (total execution time) was reduced by up to 38% for both 2 and 10 jobs filling the queue.
- Faster Jobs: For very fast jobs (e.g., eigen-100), the HQ approach was up to three times quicker in terms of makespan, despite a slight increase in CPU time due to model initiation overhead.
- Schedule Length Ratio (SLR): The SLR metric, which accounts for scheduler overhead, showed that HQ consistently outperformed SLURM, indicating better overall efficiency.
Limitations and Future Work
- Model Initiation Overhead: The current HQ implementation incurs a ~1-second overhead for initializing model servers for each job. This is noticeable for very fast jobs and can be mitigated by implementing persistent servers.
- Filesystem Dependencies: The authors encountered issues with filesystem updates for communication between the load balancer and model servers, requiring manual workarounds. They are working on a more robust network-based solution.
- Complex Workflows: The current framework is primarily tested on embarrassingly parallel tasks. Future work will focus on handling more complex workflows with interdependencies and dynamic scheduling requirements, such as adaptive GP models.
- Benchmark Diversity: The authors acknowledge that their chosen benchmarks might not be representative of all UQ workloads and plan to incorporate a more diverse mixture in future work.
Summary updated