HPC Application Cloudification: The StreamFlow Toolkit (Invited Paper)

Iacopo Colonnelli, Barbara Cantalupo, Roberto Esposito, Matteo Pennisi, Concetto Spampinato, Marco Aldinucci

TL;DR

This paper introduces StreamFlow, a novel Workflow Management System (WMS) designed to bridge the gap between traditional High-Performance Computing (HPC) and cloud environments. By enabling "hybrid workflows," StreamFlow allows computationally intensive tasks to be offloaded to HPC clusters while other parts of the workflow run on cloud infrastructure, making complex applications more accessible and efficient.

What Problem Does This Paper Address?

Accessing and utilizing High-Performance Computing (HPC) resources has traditionally been challenging, often requiring deep technical expertise in SSH-based remote shells and complex job submission mechanisms. While cloud computing has made many services more accessible, its virtualization layers can introduce overheads that are detrimental to performance-critical HPC applications. Furthermore, HPC centers are not designed for general-purpose applications, and some tasks, like providing a public web interface for data visualization, are simply not supported in their often air-gapped environments. This creates a disconnect, especially for AI applications that benefit from both the massive parallel processing power of HPC and the flexibility and accessibility of the cloud. The paper aims to solve this by proposing a "cluster-as-accelerator" pattern and a supporting WMS that can seamlessly manage workflows across these disparate environments.

Methods

The core of the paper is the StreamFlow toolkit, a Workflow Management System (WMS) built in Python 3. StreamFlow implements a "cluster-as-accelerator" design pattern, where HPC clusters act as accelerators for computationally demanding parts of a workflow. This is achieved through hybrid workflows, which allow different steps of a workflow to be scheduled on independent execution environments, including both cloud and HPC resources, without requiring shared storage.

StreamFlow integrates with the Common Workflow Language (CWL) standard for defining workflows, offering a declarative JSON or YAML syntax. It translates CWL semantics into an internal, layered dataflow model that supports broader semantics like loops and stream-based inputs. A key innovation is StreamFlow's ability to manage data movements autonomously between different executors, eliminating the need for a shared file system, which is often a barrier in hybrid cloud-HPC settings. It employs a three-layered hierarchical representation of execution environments: models (entire infrastructure), services (single agents), and resources (service instances). Connectors are used to interface with external orchestration libraries for deploying and managing these environments. The system also includes a DataManager to handle data transfers and a Scheduler to select the best resource for execution, with a default policy prioritizing data locality.

Datasets

The paper uses the BIMCV-COVID19 dataset for evaluation. This dataset is described as the largest publicly available dataset related to COVID-19 pathology, containing over 120,000 images from more than 1300 patients. The authors note that training a single variant of their pipeline on this dataset for 20 epochs takes over 15 hours on a single NVidia V100 GPU.

Key Insights and Ideas

  • Hybrid Workflows as a Paradigm Shift: The paper champions hybrid workflows as a natural way to combine the strengths of HPC and cloud computing, moving beyond the limitations of single-environment execution.
  • Cluster-as-Accelerator: This design pattern is crucial for making HPC resources more accessible by treating them as specialized accelerators for specific, compute-intensive tasks within a larger workflow.
  • Decoupling Execution Environments: StreamFlow's ability to manage workflows across heterogeneous environments without shared storage is a significant technical achievement, overcoming a common hurdle in hybrid setups.
  • Automated Data Management: The autonomous handling of data movements by StreamFlow is key to simplifying the user experience and enabling efficient execution on distributed resources.
  • Bridging AI and HPC: The work demonstrates a practical path for integrating AI workloads, which often require both massive GPU power (HPC) and flexible data handling/visualization (cloud), into a unified execution framework.

Key Results

The paper presents a case study using the CLAIRE COVID-19 universal pipeline, a complex workflow designed to compare state-of-the-art deep learning models for COVID-19 diagnosis from CT scans. This pipeline has 990 variants. The authors report that StreamFlow was used to manage the execution of this pipeline, offloading training steps to an HPC facility and collecting results on a cloud host for visualization. Preliminary results from analyzing about 1% of the pipeline variants on the HPC4AI system showed that the generated models achieved excellent accuracy, with sensitivity and specificity metrics over 90% in the best cases.

Limitations and Future Work

The authors acknowledge that the experiments presented are preliminary, with only a fraction of the pipeline variants analyzed. The main goal for future work is to continue executing all variants on the complete dataset and to establish a comprehensive framework where StreamFlow can manage pipeline execution across different hybrid HPC infrastructures. They also mention plans for integrating additional workflow design tools and formats beyond CWL, indicating a desire to avoid overly tight coupling between CWL logic and the StreamFlow runtime.

Summary updated

Abstract

Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud infrastructure can be offloaded to HPC environments to speed them up. We introduce StreamFlow, a novel Workflow Management System that supports such a design pattern and makes it possible to run the steps of a standard workflow model on independent processing elements with no shared storage. We validated the proposed approach’s effectiveness on the CLAIRE COVID-19 universal pipeline, i.e. a reproducible workflow capable of automating the comparison of (possibly all) state-of-the-art pipelines for the diagnosis of COVID-19 interstitial pneumonia from CT scans images based on Deep Neural Networks (DNNs).
BibTeX
@inproceedings{colonnelli2021hpc,
  author = {Iacopo Colonnelli and Barbara Cantalupo and Roberto Esposito and Matteo Pennisi and Concetto Spampinato and Marco Aldinucci},
  title = {{HPC Application Cloudification: The StreamFlow Toolkit (Invited Paper)}},
  booktitle = {PARMA-DITAM@HiPEAC},
  year = {2021},
  doi = {10.4230/oasics.parma-ditam.2021.5},
}
Figure