Reproducing Scientific Experiment with Cloud DevOps

Feng Zhao, Xingzhi Niu, Shao-Lun Huang, Lin Zhang

Abstract

The reproducibility of scientific experiment is vital for the advancement of disciplines based on previous work. To achieve this goal, many researchers focus on complex methodology and self-invented tools which have difficulty in practical usage. In this article, we introduce the Cloud DevOps infrastructure from software engineering community and shows how it can be used effectively for heterogeneous agents to reproduce experiments for computer science related disciplines. DevOps can be enabled using freely available cloud computing machines for medium-sized experiment and self-hosted computing engines for large-scale computing, thus powering researchers to share their experiment result with others in a more reliable way.

Introduction

Reproducing a scientific experiment is often a nightmare. Even in computer science, a result might depend on a specific version of a library, a particular OS quirk, or specialized hardware. Traditional approaches to solving this—such as custom scripts, specialized platforms, or purely methodological guidelines—often fall short because they are difficult to maintain, expensive, or too abstract for practical implementation.

This paper proposes leveraging Cloud DevOps infrastructure—the same Continuous Integration and Continuous Deployment (CI/CD) tools used in professional software engineering—to bridge this gap. By utilizing automated pipelines and "environment-as-code," researchers can ensure their experiments run reliably across diverse hardware architectures, from public cloud VMs to local high-performance clusters.

Methodology: The DevOps Framework

The core of the proposed approach lies in adopting software engineering standards for scientific workflows, specifically focusing on Infrastructure as Code and automated execution.

System Architecture

The architecture consists of three primary components: a Controller (e.g., GitHub Actions, Azure Pipelines), a Repository (e.g., GitHub, GitLab), and Agents (runners).

The interaction between the DevOps controller, code repository, and execution agent.

As shown in the figure above, the workflow is triggered by a notification from the repository to the controller. The controller then triggers an agent, which fetches the code and executes the defined pipelines. Finally, logs and artifacts are uploaded back to the controller for public access and verification. This creates an immutable audit trail where every result is tied to a specific git commit.

Automated Pipelines

By using YAML configuration files, researchers can define the exact environment (e.g., Ubuntu 16.04, Python 3.6). The automated pipeline handles the entire lifecycle of an experiment, ensuring that the process is transparent and version-controlled.

The stages of an automated scientific pipeline within a DevOps framework.

The pipeline typically follows a sequence: installing dependencies, building the application, testing, deploying to the execution environment, running the experiment, and reporting results. This structured flow ensures that the "1-to-1" correspondence between code and results is maintained.

Case Studies and Results

The flexibility of the DevOps approach was validated through several computational tasks demonstrating scalability and cross-architecture support:

  • Scalability in Graph Computing: A triangle-counting algorithm was tested on small graphs using public cloud runners (Travis CI) and then seamlessly transitioned to a self-hosted HPC node (56 cores, 256 GB RAM) for large-scale datasets, proving that the cloud can manage the workflow while local machines do the heavy lifting.
  • Serverless Bioinformatics: A protein sequence alignment experiment utilized Azure Pipelines to trigger AWS Lambda functions, completing the task in just two minutes with fully public logs.
  • Hardware Tracking: The authors demonstrated hardware tracking by running a compilation task on an ARM-based Raspberry Pi 3B managed via GitHub Actions, proving the system's ability to handle non-standard or "edge" hardware.

Discussion and Conclusion

While DevOps tools offer a robust solution for reproducibility, challenges remain regarding the "ease of use" for researchers without a systems administration background. Furthermore, scientific research cycles can be more complex than linear software build cycles, requiring more sophisticated workflow management in the future.

In conclusion, Cloud DevOps provides a highly available and scalable infrastructure for scientific reproducibility. By combining the convenience of cloud-based management with the power of local or grid-based computing, researchers can share their results in a more reliable and transparent manner.

Summary updated

Abstract

The reproducibility of scientific experiment is vital for the advancement of disciplines based on previous work. To achieve this goal, many researchers focus on complex methodology and self-invented tools which have difficulty in practical usage. In this article, we introduce the Cloud DevOps infrastructure from software engineering community and shows how it can be used effectively for heterogeneous agents to reproduce experiments for computer science related disciplines. DevOps can be enabled using freely available cloud computing machines for medium-sized experiment and self-hosted computing engines for large-scale computing, thus powering researchers to share their experiment result with others in a more reliable way.
BibTeX
@inproceedings{zhao2020reproducing,
  author = {Feng Zhao and Xingzhi Niu and Shao-Lun Huang and Lin Zhang},
  title = {{Reproducing Scientific Experiment with Cloud DevOps}},
  booktitle = {SERVICES},
  year = {2020},
  url = {https://openreview.net/forum?id=fbBAX3fjw-I},
}
Figure