TL;DR
This paper introduces StreamFlow, a novel Workflow Management System (WMS) designed to bridge the gap between traditional High-Performance Computing (HPC) and cloud environments. By enabling "hybrid workflows," StreamFlow allows computationally intensive tasks to be offloaded to HPC clusters while other parts of the workflow run on cloud infrastructure, making complex applications more accessible and efficient.
What Problem Does This Paper Address?
Accessing and utilizing High-Performance Computing (HPC) resources has traditionally been challenging, often requiring deep technical expertise in SSH-based remote shells and complex job submission mechanisms. While cloud computing has made many services more accessible, its virtualization layers can introduce overheads that are detrimental to performance-critical HPC applications. Furthermore, HPC centers are not designed for general-purpose applications, and some tasks, like providing a public web interface for data visualization, are simply not supported in their often air-gapped environments. This creates a disconnect, especially for AI applications that benefit from both the massive parallel processing power of HPC and the flexibility and accessibility of the cloud. The paper aims to solve this by proposing a "cluster-as-accelerator" pattern and a supporting WMS that can seamlessly manage workflows across these disparate environments.
Methods
The core of the paper is the StreamFlow toolkit, a Workflow Management System (WMS) built in Python 3. StreamFlow implements a "cluster-as-accelerator" design pattern, where HPC clusters act as accelerators for computationally demanding parts of a workflow. This is achieved through hybrid workflows, which allow different steps of a workflow to be scheduled on independent execution environments, including both cloud and HPC resources, without requiring shared storage.
StreamFlow integrates with the Common Workflow Language (CWL) standard for defining workflows, offering a declarative JSON or YAML syntax. It translates CWL semantics into an internal, layered dataflow model that supports broader semantics like loops and stream-based inputs. A key innovation is StreamFlow's ability to manage data movements autonomously between different executors, eliminating the need for a shared file system, which is often a barrier in hybrid cloud-HPC settings. It employs a three-layered hierarchical representation of execution environments: models (entire infrastructure), services (single agents), and resources (service instances). Connectors are used to interface with external orchestration libraries for deploying and managing these environments. The system also includes a DataManager to handle data transfers and a Scheduler to select the best resource for execution, with a default policy prioritizing data locality.
Datasets
The paper uses the BIMCV-COVID19 dataset for evaluation. This dataset is described as the largest publicly available dataset related to COVID-19 pathology, containing over 120,000 images from more than 1300 patients. The authors note that training a single variant of their pipeline on this dataset for 20 epochs takes over 15 hours on a single NVidia V100 GPU.
Key Insights and Ideas
- Hybrid Workflows as a Paradigm Shift: The paper champions hybrid workflows as a natural way to combine the strengths of HPC and cloud computing, moving beyond the limitations of single-environment execution.
- Cluster-as-Accelerator: This design pattern is crucial for making HPC resources more accessible by treating them as specialized accelerators for specific, compute-intensive tasks within a larger workflow.
- Decoupling Execution Environments: StreamFlow's ability to manage workflows across heterogeneous environments without shared storage is a significant technical achievement, overcoming a common hurdle in hybrid setups.
- Automated Data Management: The autonomous handling of data movements by StreamFlow is key to simplifying the user experience and enabling efficient execution on distributed resources.
- Bridging AI and HPC: The work demonstrates a practical path for integrating AI workloads, which often require both massive GPU power (HPC) and flexible data handling/visualization (cloud), into a unified execution framework.
Key Results
The paper presents a case study using the CLAIRE COVID-19 universal pipeline, a complex workflow designed to compare state-of-the-art deep learning models for COVID-19 diagnosis from CT scans. This pipeline has 990 variants. The authors report that StreamFlow was used to manage the execution of this pipeline, offloading training steps to an HPC facility and collecting results on a cloud host for visualization. Preliminary results from analyzing about 1% of the pipeline variants on the HPC4AI system showed that the generated models achieved excellent accuracy, with sensitivity and specificity metrics over 90% in the best cases.
Limitations and Future Work
The authors acknowledge that the experiments presented are preliminary, with only a fraction of the pipeline variants analyzed. The main goal for future work is to continue executing all variants on the complete dataset and to establish a comprehensive framework where StreamFlow can manage pipeline execution across different hybrid HPC infrastructures. They also mention plans for integrating additional workflow design tools and formats beyond CWL, indicating a desire to avoid overly tight coupling between CWL logic and the StreamFlow runtime.
Summary updated