SCHeMa: Scheduling Scientific Containers on a Cluster of Heterogeneous Machines

Thanasis Vergoulis, Konstantinos Zagganas, Loukas Kavouras, Martin Reczko, Stelios Sartzetakis, Theodore Dalamagas

Abstract

In the era of data-driven science, conducting computational experiments that involve analysing large datasets using heterogeneous computational clusters is part of the everyday routine for many scientists. Moreover, to ensure the credibility of their results, it is very important for these analyses to be easily reproducible by other researchers. Although various technologies that could facilitate the work of scientists in this direction have been introduced in recent years, there is still a lack of open-source platforms that combine them to this end. In this work, we describe and demonstrate SCHeMa, an open-source platform that facilitates the execution and reproducibility of computational analysis on heterogeneous clusters, leveraging containerization, experiment packaging, workflow management, and machine learning technologies.

Introduction

Modern data-driven science faces two primary hurdles: the reproducibility crisis and increasing hardware complexity. Replicating results often becomes a struggle against "dependency hell," where deprecated library versions or software conflicts prevent successful execution. Furthermore, research institutions often manage heterogeneous clusters—collections of machines with varying specifications (e.g., differing RAM, GPUs, or FPGAs). Inefficient scheduling on these clusters leads to significant resource waste, such as small tasks occupying high-memory nodes and blocking more demanding jobs.

System Architecture

SCHeMa (Scheduler for scientific Containers on a cluster of Heterogeneous Machines) addresses these challenges by integrating containerization, workflow management, and machine learning into a unified platform. The architecture consists of several internal components that interface with a Kubernetes-managed heterogeneous cluster.

High-level architecture of the SCHeMa platform, illustrating internal components and their interaction with the heterogeneous cluster.

Key components include:

  • Job Submission API & User Interface: For seamless user interaction.
  • Software Profiler & Job Classifier: The core intelligence for automated scheduling.
  • Experiment Packager: Bundles code, data, and configuration using the RO-crate specification.
  • Data Repository Connector: Facilitates interaction with third-party data repositories like Zenodo.

Intelligent Scheduling and Profiling

The Software Profiler uses machine learning (scikit-learn) to automate scheduling decisions. When a new tool is uploaded, the profiler executes it multiple times with varying inputs to monitor resource consumption. It then trains a classification model to predict the optimal node type (e.g., high-memory vs. regular) based on the metadata of the input parameters. This ensures that jobs are routed to the most efficient hardware, preserving high-spec resources for tasks that truly require them.

User Experience and Reproducibility

SCHeMa prioritizes accessibility through a web-based UI that automatically generates input forms based on Common Workflow Language (CWL) descriptions. This allows scientists to run complex Docker-based workflows without requiring deep command-line expertise.

The SCHeMa web interface, showing software management and the automated form generation for job submission.

To ensure reproducibility, SCHeMa leverages Docker for environment consistency and CWL for workflow definition. The platform facilitates the "afterlife" of an experiment by allowing users to publish finished runs as citable RO-crates, preventing vendor lock-in and promoting open science.

Experimental Evaluation

The platform was demonstrated on the ELIXIR-GR Cloud Infrastructure, a real-world cluster consisting of 45 physical nodes, 2,600 CPU cores, and 24 TB of RAM. The system utilized over 2,000 ready-to-use scientific containers from Biocontainers. Results showed that the ML-based Job Classifier effectively prevents resource waste by correctly routing jobs based on predicted needs, keeping high-spec hardware available for demanding tasks.

Conclusion and Future Work

SCHeMa successfully bridges the gap between complex scientific workflows and heterogeneous computing. While the initial profiling phase requires a compute investment to build the ML models, it provides significant long-term value for frequently used pipelines. Future work will focus on expanding support for high-performance file systems beyond the current NFS implementation.

Summary updated

Abstract

In the era of data-driven science, conducting computational experiments that involve analysing large datasets using heterogeneous computational clusters, is part of the everyday routine for many scientists. Moreover, to ensure the credibility of their results, it is very important for these analyses to be easily reproducible by other researchers. Although various technologies, that could facilitate the work of scientists in this direction, have been introduced in the recent years, there is still a lack of open source platforms that combine them to this end. In this work, we describe and demonstrate SCHeMa, an open-source platform that facilitates the execution and reproducibility of computational analysis on heterogeneous clusters, leveraging containerization, experiment packaging, workflow management, and machine learning technologies.
BibTeX
@inproceedings{vergoulis2021schema,
  author = {Thanasis Vergoulis and Konstantinos Zagganas and Loukas Kavouras and Martin Reczko and Stelios Sartzetakis and Theodore Dalamagas},
  title = {{SCHeMa: Scheduling Scientific Containers on a Cluster of Heterogeneous Machines}},
  booktitle = {SSDBM},
  year = {2021},
  url = {https://openreview.net/forum?id=MR8F1U1gl_},
}
Figure