Melissa: coordinating large-scale ensemble runs for deep learning and sensitivity analyses

Marc Schouler, Robert Alexander Caulk, Lucas Thibaut Meyer, Théophile Terraz, Christoph Conrads, Sebastian Friedemann, Achal Agarwal, Juan Manuel Baldonado, Bartlomiej Pogodzinski, Anna Sekula, Alejandro Ribés, Bruno Raffin, Bartłomiej Pogodziński, Anna SekułaJ. Open Source Softw 2023 · DOI · DBLP

Abstract

Melissa is an open-source, "file-avoiding" framework designed to orchestrate massive ensemble simulations on supercomputers by bypassing the traditional disk I/O bottleneck. By processing data "in-transit" directly from simulation clients to a centralized server, it enables large-scale sensitivity analyses and deep learning training that would otherwise be impossible due to storage and speed limitations.

Introduction: The HPC Storage Wall

In the field of High-Performance Computing (HPC), researchers are increasingly encountering a "storage wall." While supercomputers have become incredibly fast at processing data, the speed at which they can read and write to disks (I/O) has lagged significantly. This bottleneck is particularly problematic for "ensemble simulations"—studies involving thousands of model executions with varying parameters to train AI or perform sensitivity analyses.

Traditionally, these simulations write results to disk for later analysis, generating petabytes of data. To manage this, scientists often sacrifice accuracy by lowering resolution or downsampling data. Melissa addresses this by eliminating the need to write intermediate simulation data to disk, processing it "on-the-fly" instead.

Methodology and Architecture

Melissa utilizes a three-tier asynchronous architecture to handle data in-transit, moving it directly from the memory of simulation nodes to analysis nodes.

The three-tier asynchronous architecture of the Melissa framework, showing the interaction between the launcher, scheduler, clients, and specialized servers.

The framework consists of three primary components:

  1. Melissa Client: The user’s simulation code is instrumented with three simple API calls (init, send, and finalize). Instead of writing to a file, the simulation sends its output directly over the network to the server.
  2. Melissa Server: A parallelized process that receives data from multiple clients simultaneously. It is specialized into melissa-sa for iterative sensitivity analysis and melissa-dl for online deep learning training using a memory buffer.
  3. Melissa Launcher: A Python-based orchestrator that interacts with cluster schedulers (e.g., Slurm) to submit jobs, monitor status, and provide fault tolerance by restarting failed components.

Simultaneous Data Processing

A key advantage of Melissa is the transition from sequential to simultaneous workflows. In classical approaches, data generation must complete before training begins. Melissa allows these phases to overlap, significantly reducing the total time-to-solution and bypassing the file system entirely.

Comparison between classical sequential training and Melissa's simultaneous data generation and training approach.

Performance and Scalability

Melissa has demonstrated impressive performance in real-world large-scale scenarios, proving that bypassing the disk is mandatory for scale. It has been successfully coupled with state-of-the-art solvers such as Code-Saturne and FEniCS. Key results include:

  • Scale: Management of up to 30,000 CPU cores simultaneously.
  • Throughput: Execution of 80,000 parallel simulations in a single study.
  • Efficiency: Avoidance of 288 TB of intermediate data storage in a single study, representing massive savings in time and costs.
  • Integration: Full compatibility with modern deep learning stacks, including PyTorch, TensorFlow, and real-time monitoring via TensorBoard.

Conclusion and Future Work

While Melissa offers a robust solution for the I/O bottleneck, it requires simulation source code access for API instrumentation. Additionally, the "in-transit" method increases demand on network bandwidth and server memory. Future work will focus on refining memory management for high-velocity data and extending the library of iterative statistical algorithms available within the server framework.

Summary updated

BibTeX
@article{schouler2023melissa,
  author = {Marc Schouler and Robert Alexander Caulk and Lucas Thibaut Meyer and Théophile Terraz and Christoph Conrads and Sebastian Friedemann and Achal Agarwal and Juan Manuel Baldonado and Bartlomiej Pogodzinski and Anna Sekula and Alejandro Ribés and Bruno Raffin and Lucas Meyer and Théophile Terraz and Christoph Conrads and Sebastian Friedemann and A. K. Agarwal and Juan Manuel Baldonado and Bartłomiej Pogodziński and Anna Sekuła and Bruno Raffin},
  title = {{Melissa: coordinating large-scale ensemble runs for deep learning and sensitivity analyses}},
  journal = {J. Open Source Softw.},
  year = {2023},
  doi = {10.21105/joss.05291},
}
Figure