TL;DR
This paper introduces LowFive, a novel data transport layer for high-performance computing (HPC) workflows that enables in situ data analysis without requiring modifications to existing scientific codes. By leveraging the HDF5 Virtual Object Layer (VOL), LowFive seamlessly integrates with applications that already use HDF5, allowing them to communicate data directly in memory or via traditional file I/O, significantly improving performance and flexibility for complex scientific simulations.
What Problem Does This Paper Address?
Modern supercomputers are experiencing a growing disparity between their computational power and I/O bandwidth. This bottleneck, coupled with the increasing need to analyze massive datasets generated by simulations more frequently than traditional checkpointing allows, necessitates more efficient data handling strategies. Traditional approaches involve writing all simulation data to disk and then analyzing it separately, which is slow and inefficient. In situ workflows aim to solve this by enabling communication and analysis directly between simulation and analysis tasks while they are running, often using the HPC interconnect. However, integrating these workflows typically requires significant code modifications to handle data models and communication protocols. This paper addresses the challenge of enabling in situ data transport for HPC workflows with minimal to no changes to existing scientific simulation codes, which often rely on established data models like HDF5.
Methods
LowFive is implemented as an HDF5 Virtual Object Layer (VOL) plugin. This allows it to intercept HDF5 calls and redirect them, enabling custom data transport logic without altering the application's source code. Key features of LowFive include:
- HDF5 Data Model Integration: It leverages the HDF5 data model, a widely adopted standard in HPC, AI, and big data, providing a familiar interface for users.
- Dual Transport Modes: LowFive supports two primary modes:
- In Situ (MPI): Data is communicated directly between processes using MPI message passing.
- File-based: Data can be written to and read from traditional HDF5 files on physical storage.
- Flexible Data Handling: Users can choose between deep copies of datasets or shallow references, allowing for independent modification of local copies or efficient memory usage.
- Fan-in/Fan-out Support: Multiple producer tasks can write data, and multiple consumer tasks can read data, accommodating complex workflow task graphs.
- Data Redistribution: LowFive handles scenarios where producer and consumer tasks have different numbers of MPI processes or data decompositions, enabling efficient data movement between them. This is achieved through an "index-serve-query" mechanism that determines data intersections and facilitates direct point-to-point communication.
Datasets
The paper uses synthetic datasets for benchmarking, consisting of:
- A regular grid of 64-bit unsigned integer scalar values.
- A list of particles, each represented as a 3D vector of 32-bit floating-point values.
These datasets are designed to encode global position information, allowing for validation of data redistribution. The experiments also include a real-world use case from a high-energy physics simulation (Nyx) coupled with an analysis task (Reeber).
Key Insights and Ideas
- VOL Plugins as a Seamless Integration Strategy: Implementing LowFive as an HDF5 VOL plugin is a powerful approach for enabling in situ workflows with minimal user effort. It allows existing codes to be "plugged in" without modification, democratizing access to advanced data transport capabilities.
- The Performance Gap Between In-Memory and File I/O: The experiments clearly demonstrate that in-memory communication via MPI is orders of magnitude faster than writing to and reading from parallel file systems. This highlights the critical need for in situ solutions to avoid the I/O bottleneck.
- Overhead of LowFive is Minimal for File I/O: When operating in file mode, LowFive introduces negligible overhead compared to direct HDF5 file operations, making it a transparent enhancement.
- Data Redistribution is Crucial for Heterogeneous Workflows: The ability to handle differing process counts and data decompositions between tasks is essential for real-world workflows, and LowFive's index-serve-query mechanism effectively addresses this.
- HDF5's Rich Data Model is a Double-Edged Sword: While LowFive benefits from HDF5's expressive metadata, certain application-specific optimizations (like data repacking in Nyx) can undermine LowFive's zero-copy capabilities, forcing deep copies and increasing memory usage.
Key Results
- Significant Speedup for In Situ Workflows: In the synthetic benchmarks, LowFive's in-situ (MPI) mode was hundreds of times faster than file-based communication. At the largest scale tested (16K processes), it achieved communication of 223 GiB in just over 3 seconds.
- Comparable Performance to Pure MPI: LowFive's in-situ mode performed comparably to, and in some cases slightly better than, a hand-written pure MPI redistribution code.
- Competitive with DataSpaces: While DataSpaces was generally faster (up to 20% at the largest scale), LowFive offered advantages in terms of code modification requirements and support for the full HDF5 hierarchical data model.
- Substantial Improvement in a Real-World Use Case: In the Nyx-Reeber cosmology simulation, LowFive outperformed both standard HDF5 and AMREX's plotfile format by significant margins, achieving speedups of up to an order of magnitude. Crucially, no code modifications were needed for Nyx or Reeber.
Limitations and Future Work
- Zero-Copy Limitations: Application-level optimizations that repack data (e.g., in Nyx) can force LowFive to perform deep copies, negating its zero-copy advantage and increasing memory consumption.
- Synchronization Overhead: The paper notes that synchronization, particularly during the index-serve-query redistribution and the HDF5 API's file close signal, can impact LowFive's performance.
- Future Work: The authors plan to investigate ways to minimize synchronization, overlap reading and writing, and explore more efficient collective communication algorithms. They also aim to add support for the remaining HDF5 API functions and build a higher-level workflow system that utilizes LowFive.
Summary updated