Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments

Jonathan Bader, Jim Belak, Matthew T. Bement, Matthew Berry, Robert Carson, Daniela Cassol, Stephen Chan, John Coleman, Kastan Day, Alejandro Duque, Kjiersten Fagnan, Jeff Froula, Shantenu Jha, Daniel S. Katz, Piotr Kica, Volodymyr V. Kindratenko, Edward Kirton, Ramani Kothadia, Daniel E. Laney, Fabian Lehmann, Ulf Leser, Sabina Licholai, Maciej Malawski, Mario Melara, Elais Player Jackson, Matthew Rolchigo, Setareh Sarrafan, Seung-Jin Sul, Abdullah Syed, Lauritz Thamsen, Mikhail Titov, Matteo Turilli, Silvina Caíno-Lores, Anirban MandalWorkshops @ SC 2023 · DOI

Abstract

Scientific workflows often suffer from significant inefficiencies due to a lack of communication between Scientific Workflow Management Systems (SWMS) and underlying resource managers. This paper introduces the Common Workflow Scheduler Interface (CWSI), a standardized bridge that enables "workflow-aware" scheduling. By sharing task dependencies and resource requirements with resource managers like Kubernetes, the proposed system achieves a median runtime improvement of 24.8% across various bioinformatics workflows. This work demonstrates that providing resource managers with the context of the workflow graph can substantially reduce makespan and improve resource utilization in high-performance computing environments.

Introduction

In high-performance computing, researchers face a persistent "M x N" integration problem. Scientists utilize a variety of Scientific Workflow Management Systems (SWMS), such as Nextflow or Apache Airflow (the "M"), which must be executed across diverse resource managers like SLURM or Kubernetes (the "N").

Currently, these layers operate in relative isolation. Resource managers typically treat every task as an independent black box, unaware of the larger Directed Acyclic Graph (DAG) to which it belongs. This lack of "workflow awareness" leads to inefficiencies: resource managers may schedule tasks in an order that creates bottlenecks or over-allocate resources to tasks waiting on predecessors. Building custom integrations for every combination of SWMS and resource manager is a significant maintenance burden, often leaving advanced scheduling features unsupported.

Methods: The CWSI Framework

To address these challenges, the authors propose the Common Workflow Scheduler Interface (CWSI) and the Common Workflow Scheduler (CWS).

The CWSI architecture acts as a bridge between SWMS and resource managers, enabling workflow-aware scheduling.
  • The Interface (CWSI): A standardized API that allows an SWMS to pass essential metadata—including input file details, CPU/memory requests, and task dependencies—to the resource manager.
  • The Scheduler (CWS): A component residing within the resource manager that stores the workflow graph in memory to inform scheduling decisions.
  • Plugin Architecture: The authors developed a Nextflow plugin to translate internal logic into CWSI calls and implemented a CWS component for Kubernetes.
  • Workflow-Aware Scheduling: The system utilizes a "Rank (Min) Round Robin" strategy. Unlike standard FIFO or simple Round Robin scheduling, this strategy leverages the workflow's structure to prioritize tasks on the critical path.

Evaluation and Results

The system was evaluated using the nf-core suite, a collection of curated bioinformatics workflows. Nine popular workflows, including AmpliSeq (phylogenetics), RNA-Seq (gene expression), Sarek (variant calling), and MAG (metagenomics), were tested using standard datasets to compare the original Kubernetes scheduling strategy against the CWSI-enabled approach.

Runtime performance comparison showing significant reductions in execution time for the Rank (Min) Round Robin strategy across multiple workflows.

The prototype implementation demonstrated significant performance gains:

  • Speedup: The "Rank (Min) Round Robin" strategy achieved a median runtime improvement of 24.8% compared to standard Nextflow-Kubernetes interactions.
  • Efficiency: The total makespan (completion time) was reduced by an average of 10.8%.
  • Consistency: As shown in Figure 2, the workflow-aware strategy consistently outperformed the original strategy across almost all tested workflows, often with reduced variance in execution time.

Discussion and Future Work

The core insight of this research is that resource managers do not necessarily require complex AI to improve performance; they simply require context. By exposing the DAG to the scheduler, the system can identify and prioritize bottleneck tasks. Furthermore, because CWSI sits at the intersection of software and hardware, it serves as an ideal point for collecting provenance data, facilitating scientific reproducibility.

Future work aims to expand the ecosystem beyond Nextflow and Kubernetes to include Airflow and Argo. The authors are also exploring predictive models like Lotaru to estimate runtimes for new workflows and Tarema to dynamically match tasks to the most suitable nodes in hyper-heterogeneous environments where hardware capabilities vary significantly.

Summary updated

BibTeX
@inproceedings{bader2023novel,
  author = {Jonathan Bader and Jim Belak and Matthew T. Bement and Matthew Berry and Robert Carson and Daniela Cassol and Stephen Chan and John Coleman and Kastan Day and Alejandro Duque and Kjiersten Fagnan and Jeff Froula and Shantenu Jha and Daniel S. Katz and Piotr Kica and Volodymyr V. Kindratenko and Edward Kirton and Ramani Kothadia and Daniel E. Laney and Fabian Lehmann and Ulf Leser and Sabina Licholai and Maciej Malawski and Mario Melara and Elais Player Jackson and Matthew Rolchigo and Setareh Sarrafan and Seung-Jin Sul and Abdullah Syed and Lauritz Thamsen and Mikhail Titov and Matteo Turilli and Silvina Caíno-Lores and Anirban Mandal},
  title = {{Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments}},
  booktitle = {Workshops @ SC},
  year = {2023},
  doi = {10.1145/3624062.3626283},
}
Figure