Abstract
Melissa is an open-source, "file-avoiding" framework designed to orchestrate massive ensemble simulations on supercomputers by bypassing the traditional disk I/O bottleneck. By processing data "in-transit" directly from simulation clients to a centralized server, it enables large-scale sensitivity analyses and deep learning training that would otherwise be impossible due to storage and speed limitations.
Introduction: The HPC Storage Wall
In the field of High-Performance Computing (HPC), researchers are increasingly encountering a "storage wall." While supercomputers have become incredibly fast at processing data, the speed at which they can read and write to disks (I/O) has lagged significantly. This bottleneck is particularly problematic for "ensemble simulations"—studies involving thousands of model executions with varying parameters to train AI or perform sensitivity analyses.
Traditionally, these simulations write results to disk for later analysis, generating petabytes of data. To manage this, scientists often sacrifice accuracy by lowering resolution or downsampling data. Melissa addresses this by eliminating the need to write intermediate simulation data to disk, processing it "on-the-fly" instead.
Methodology and Architecture
Melissa utilizes a three-tier asynchronous architecture to handle data in-transit, moving it directly from the memory of simulation nodes to analysis nodes.
The framework consists of three primary components:
- Melissa Client: The user’s simulation code is instrumented with three simple API calls (
init,send, andfinalize). Instead of writing to a file, the simulation sends its output directly over the network to the server. - Melissa Server: A parallelized process that receives data from multiple clients simultaneously. It is specialized into
melissa-safor iterative sensitivity analysis andmelissa-dlfor online deep learning training using a memory buffer. - Melissa Launcher: A Python-based orchestrator that interacts with cluster schedulers (e.g., Slurm) to submit jobs, monitor status, and provide fault tolerance by restarting failed components.
Simultaneous Data Processing
A key advantage of Melissa is the transition from sequential to simultaneous workflows. In classical approaches, data generation must complete before training begins. Melissa allows these phases to overlap, significantly reducing the total time-to-solution and bypassing the file system entirely.
Performance and Scalability
Melissa has demonstrated impressive performance in real-world large-scale scenarios, proving that bypassing the disk is mandatory for scale. It has been successfully coupled with state-of-the-art solvers such as Code-Saturne and FEniCS. Key results include:
- Scale: Management of up to 30,000 CPU cores simultaneously.
- Throughput: Execution of 80,000 parallel simulations in a single study.
- Efficiency: Avoidance of 288 TB of intermediate data storage in a single study, representing massive savings in time and costs.
- Integration: Full compatibility with modern deep learning stacks, including PyTorch, TensorFlow, and real-time monitoring via TensorBoard.
Conclusion and Future Work
While Melissa offers a robust solution for the I/O bottleneck, it requires simulation source code access for API instrumentation. Additionally, the "in-transit" method increases demand on network bandwidth and server memory. Future work will focus on refining memory management for high-velocity data and extending the library of iterative statistical algorithms available within the server framework.
Summary updated