Abstract
Scientific workflows often suffer from significant inefficiencies due to a lack of communication between Scientific Workflow Management Systems (SWMS) and underlying resource managers. This paper introduces the Common Workflow Scheduler Interface (CWSI), a standardized bridge that enables "workflow-aware" scheduling. By sharing task dependencies and resource requirements with resource managers like Kubernetes, the proposed system achieves a median runtime improvement of 24.8% across various bioinformatics workflows. This work demonstrates that providing resource managers with the context of the workflow graph can substantially reduce makespan and improve resource utilization in high-performance computing environments.
Introduction
In high-performance computing, researchers face a persistent "M x N" integration problem. Scientists utilize a variety of Scientific Workflow Management Systems (SWMS), such as Nextflow or Apache Airflow (the "M"), which must be executed across diverse resource managers like SLURM or Kubernetes (the "N").
Currently, these layers operate in relative isolation. Resource managers typically treat every task as an independent black box, unaware of the larger Directed Acyclic Graph (DAG) to which it belongs. This lack of "workflow awareness" leads to inefficiencies: resource managers may schedule tasks in an order that creates bottlenecks or over-allocate resources to tasks waiting on predecessors. Building custom integrations for every combination of SWMS and resource manager is a significant maintenance burden, often leaving advanced scheduling features unsupported.
Methods: The CWSI Framework
To address these challenges, the authors propose the Common Workflow Scheduler Interface (CWSI) and the Common Workflow Scheduler (CWS).
- The Interface (CWSI): A standardized API that allows an SWMS to pass essential metadata—including input file details, CPU/memory requests, and task dependencies—to the resource manager.
- The Scheduler (CWS): A component residing within the resource manager that stores the workflow graph in memory to inform scheduling decisions.
- Plugin Architecture: The authors developed a Nextflow plugin to translate internal logic into CWSI calls and implemented a CWS component for Kubernetes.
- Workflow-Aware Scheduling: The system utilizes a "Rank (Min) Round Robin" strategy. Unlike standard FIFO or simple Round Robin scheduling, this strategy leverages the workflow's structure to prioritize tasks on the critical path.
Evaluation and Results
The system was evaluated using the nf-core suite, a collection of curated bioinformatics workflows. Nine popular workflows, including AmpliSeq (phylogenetics), RNA-Seq (gene expression), Sarek (variant calling), and MAG (metagenomics), were tested using standard datasets to compare the original Kubernetes scheduling strategy against the CWSI-enabled approach.
The prototype implementation demonstrated significant performance gains:
- Speedup: The "Rank (Min) Round Robin" strategy achieved a median runtime improvement of 24.8% compared to standard Nextflow-Kubernetes interactions.
- Efficiency: The total makespan (completion time) was reduced by an average of 10.8%.
- Consistency: As shown in Figure 2, the workflow-aware strategy consistently outperformed the original strategy across almost all tested workflows, often with reduced variance in execution time.
Discussion and Future Work
The core insight of this research is that resource managers do not necessarily require complex AI to improve performance; they simply require context. By exposing the DAG to the scheduler, the system can identify and prioritize bottleneck tasks. Furthermore, because CWSI sits at the intersection of software and hardware, it serves as an ideal point for collecting provenance data, facilitating scientific reproducibility.
Future work aims to expand the ecosystem beyond Nextflow and Kubernetes to include Airflow and Argo. The authors are also exploring predictive models like Lotaru to estimate runtimes for new workflows and Tarema to dynamically match tasks to the most suitable nodes in hyper-heterogeneous environments where hardware capabilities vary significantly.
Summary updated