PyExperimenter: Easily distribute experiments and track results

Tanja Tornede, Alexander Tornede, Lukas Fehring, Lukas Gehring, Helena Graf, Jonas Hanselle, Felix Mohr, Marcel WeverJ. Open Source Softw 2023 · DOI · DBLP

Abstract

PyExperimenter is an open-source Python library designed to automate the execution and tracking of large-scale empirical studies. By utilizing a central database and a pull-based workflow, it simplifies the distribution of experiments across high-performance computing (HPC) clusters, eliminating the need for manual configuration management and fragmented data collection. The tool provides a lightweight alternative to enterprise MLOps platforms, focusing on ease of use and integration with the standard data science stack.

Introduction

In fields such as Artificial Intelligence, evaluating algorithms across hundreds or thousands of parameter combinations is a standard yet cumbersome task. Researchers often face "configuration hell," managing individual files for every run, and "distributed chaos," manually tracking job success or failure on high-performance computing (HPC) systems. Furthermore, "data fragmentation" occurs when results must be scraped from disparate log files.

While heavyweight MLOps tools like Weights & Biases or MLFlow exist, they typically employ a "push" model that can be difficult to manage at scale and may lock users into specific platforms. PyExperimenter addresses these challenges by providing a streamlined, database-centric approach to experiment management, specifically designed for the needs of researchers.

Methods

PyExperimenter introduces an inverted, pull-based workflow to simplify large-scale evaluations. The process begins with the user defining an "experiment grid" or search space in a single configuration file. This grid is used to populate a central database table where every experiment combination is initially marked as "open."

The PyExperimenter workflow: Users specify grids, a central database manages experiment states, and HPC workers pull and execute jobs.

Rather than a central orchestrator pushing jobs, worker instances are launched on local or HPC resources. These workers automatically "pull" the next available experiment from the database, execute the user-defined Python function, and write results or error logs directly back to the database. This architecture offers several advantages:

  • HPC Compatibility: It is easier to launch identical worker scripts that request work than to manage unique scripts with hardcoded parameters.
  • Source of Truth: The database serves as the central hub for both inputs and outputs, eliminating intermediate log files.
  • Fault Tolerance: Crashed experiments remain "open" or are marked as "failed," allowing for easy resets and recovery from hardware issues.
  • Transparency: Users can query or modify experiment statuses using standard SQL or Python, providing more transparency than "black-box" tracking tools.

Results

PyExperimenter serves as a lightweight, open-source alternative to enterprise platforms. It significantly reduces the manual effort required for configuration generation and result aggregation. The tool has demonstrated the ability to handle massively parallel evaluations on HPC systems while maintaining ease of use through a single configuration file and a standard Python interface. Furthermore, its integration with the modern data science stack allows results to be exported directly into Pandas DataFrames for immediate statistical analysis and visualization.

Discussion and Limitations

While PyExperimenter is highly effective for grid-centric experiments, it may be less intuitive for dynamic or evolutionary optimization strategies. The authors position it as a "less heavyweight" alternative to tools like ClearML or Polyaxon, specifically tailored for researchers who prioritize open-source flexibility and direct database access. Future developments may include enhanced support for complex database backends and deeper integration with specific HPC schedulers like Slurm.

Conclusion

By automating the "manual labor" of empirical studies, PyExperimenter allows researchers to focus on analysis rather than infrastructure management. Its pull-based model and database-centric design provide a robust framework for reproducible and scalable research in computational fields.

Summary updated

BibTeX
@article{tornede2023pyexperimenter,
  author = {Tanja Tornede and Alexander Tornede and Lukas Fehring and Lukas Gehring and Helena Graf and Jonas Hanselle and Felix Mohr and Marcel Wever and Lukas Fehring and Lukas Gehring and Helena Graf and Jonas Hanselle},
  title = {{PyExperimenter: Easily distribute experiments and track results}},
  journal = {J. Open Source Softw.},
  year = {2023},
  doi = {10.21105/joss.05149},
}
Figure