TL;DR
Accurately simulating High-Performance Computing (HPC) storage systems requires realistic I/O workloads. This paper introduces IOWA, a flexible abstraction layer that unifies diverse I/O workload generation methods (traces, synthetic kernels, characterizations), allowing researchers to choose the best approach for their specific evaluation needs.
What Problem Does This Paper Address?
Evaluating HPC storage system designs hinges on using I/O workloads that truly reflect real-world usage. However, existing workload modeling techniques like synthetic benchmarks or trace replay have limitations, and no single method is universally applicable. Researchers need a way to generate or replay I/O patterns that are representative of complex HPC applications, but they face challenges in capturing the nuances of these workloads, especially for large-scale, coordinated I/O. This paper aims to provide a flexible framework that bridges this gap, enabling more accurate and adaptable I/O workload modeling for storage system analysis.
Methods
The core contribution is IOWA (I/O Workload Abstraction), a modular framework designed to decouple workload generators from consumers. IOWA provides a unified API that allows various workload consumers (like storage simulators or replay tools) to ingest I/O workloads from different sources.
The paper details three distinct workload generation methods that interface with IOWA:
- I/O Trace Workloads (using Recorder): This method uses Recorder, a multilevel tracing framework, to capture detailed I/O operations (including timing and parameters) at various layers of the I/O stack (POSIX, MPI-IO). This allows for high-fidelity reproduction of application I/O patterns.
- Synthetic I/O Workloads (using CODES I/O Language): This approach leverages the CODES I/O language, a domain-specific language for describing I/O patterns. It allows for the creation of arbitrary synthetic workloads, offering flexibility for exploring hypothetical scenarios.
- I/O Characterization Workloads (using Darshan): This method utilizes Darshan, a lightweight tool that collects high-level I/O statistics (access sizes, interarrival times) rather than full traces. The paper describes heuristics to transform these compact characterizations into representative I/O workloads.
IOWA itself is designed with specific criteria: it models workloads as ordered sets of processes (ranks), includes delays and synchronization points, supports independent I/O streams per process, and offers an "undo" mechanism for compatibility with optimistic simulation systems. The paper focuses on modeling I/O at the POSIX layer for broad portability.
Datasets
The paper uses two primary application I/O workloads for evaluation:
- VPIC-IO: An I/O kernel from the VPIC plasma physics simulation code, which uses H5Part for storing time-varying datasets.
- IOR benchmark: A widely used HPC I/O benchmark, configured for both file-per-process and shared-file workloads.
The data sources for these workloads were generated using:
- Recorder traces: Obtained by running VPIC-IO and IOR with link-time instrumentation on the Mira supercomputer.
- CODES I/O language descriptions: Manually crafted for VPIC-IO and IOR.
- Darshan logs: Collected from VPIC-IO and IOR runs on Mira.
The paper also references the ALCF I/O Data Repository as a source for Darshan characterizations of numerous HPC jobs.
Key Insights and Ideas
- No single best workload model: The paper strongly emphasizes that different I/O workload generation techniques have distinct trade-offs in accuracy, flexibility, and breadth of use. The choice depends heavily on the specific evaluation goals.
- Abstraction is key: IOWA's unified abstraction layer is crucial for enabling researchers to seamlessly switch between different workload generation methods and consumers, promoting flexibility and reusability.
- High-level I/O is tricky: Accurately modeling collective I/O operations and the internal traffic of high-level libraries (like HDF5 metadata) is a significant challenge. These subtle aspects can significantly impact performance and are not always captured by lower-level traces or simplified characterizations.
- Topology matters for replay: When replaying workloads on real systems, understanding platform-specific topology and optimizations (e.g., aggregator selection in MPI-IO) is critical for accurate performance prediction.
- Memory allocation impacts performance: Even with accurate I/O patterns, differences in memory buffer allocation strategies between the original application and the replay tool can lead to significant performance discrepancies.
Key Results
- Storage System Simulation: When simulating the VPIC-IO workload, all three generators (Recorder, CODES, Darshan) showed similar aggregate write rates, but subtle differences in write request rates led to slight execution time variations. The Darshan generator, while simpler, did not capture periodic dips in write rates as well as Recorder or CODES.
- I/O Workload Replay:
- For a simple IOR file-per-process workload, all three generators tracked the original workload's runtime closely (within 10% error).
- For a more complex IOR shared-file workload involving collective I/O, the Recorder generator performed best, staying within 5% error at modest scales. However, at larger scales (128K processes), it showed a ~26% decrease in runtime compared to the target. The CODES and Darshan generators performed significantly worse, with runtime increases of over 55% at 128K processes. This disparity was attributed to their inability to capture platform-specific topology optimizations for collective I/O.
- Workload Size: CODES I/O language and Darshan datasets were significantly smaller (orders of magnitude) than Recorder traces, making them more suitable for storage and collaboration.
Limitations and Future Work
- Limitations:
- The paper acknowledges that modeling collective I/O accurately, especially with topology-aware optimizations, is challenging for synthetic and characterization-based methods.
- The Recorder generator, while accurate, can reproduce anomalies like straggling processes, leading to performance variations.
- The IOWA POSIX-level modeling might not fully capture the nuances of high-level I/O libraries.
- Determining an application's specific memory allocation strategy for accurate replay can be difficult.
- Future Work:
- Integrate more workload generation techniques into IOWA, including those using probability distributions and extrapolation methods.
- Refine the Darshan synthesis technique by incorporating more detailed application I/O workload information.
- Investigate alternative distribution functions (e.g., Poisson, Pareto) for modeling I/O request rates from Darshan data.
- Explore ways to better model high-level I/O operations and their underlying implementations, potentially by extending IOWA or integrating with other analysis tools.
- Further investigate the impact of memory allocation strategies on I/O performance.
Summary updated