Characterizing Machine Learning I/O Workloads on Leadership Scale HPC Systems

Arnab Kumar Paul, Ahmad Maroof Karimi, Feiyi WangMASCOTS 2021 · OpenReview

Abstract

High performance computing (HPC) is no longer solely limited to traditional workloads such as simulation and modeling. With the increase in the popularity of machine learning (ML) and deep learning (DL) technologies, an increasing number of HPC users are incorporating ML methods into their workflow and scientific discovery processes. This gives rise to a diverse set of I/O patterns compared to traditional checkpoint/restart-based HPC behavior. This paper provides an in-depth analysis of ML I/O behavior using Darshan logs from over 23,000 jobs on the Summit supercomputer over one year. We characterize how I/O behavior differs across science domains and analyze the usage of parallel file systems versus burst buffers.

Introduction

Traditionally, High-Performance Computing (HPC) was dominated by massive simulations that are "write-heavy," saving large checkpoints to stable storage. However, the rise of Machine Learning (ML) and Deep Learning (DL) has shifted the landscape. ML workloads are typically "read-intensive," involving millions of small files and random access patterns. Despite this shift, there is a lack of holistic understanding regarding ML workload behavior at the "leadership scale" (thousands of nodes). This study bridges that gap by analyzing Darshan profiling data from the Summit supercomputer to provide a roadmap for designing storage systems for next-generation AI-driven science.

Methodology

To extract ML-specific data from general HPC logs, a multi-step pipeline was developed:

  1. Keyword Annotation: A list of 42 keywords (e.g., pytorch, tensorflow, horovod) was used to scan Darshan logs for ML-related executables and files.
  2. Data Integration: Darshan logs were merged with Summit’s scheduler logs to associate jobs with science domains and node counts.
  3. Classification: Jobs were categorized by scale (Small, Medium, Flagship) and I/O type (Read-Intensive, Write-Intensive, or Read-Write).
  4. Storage Comparison: Performance was compared between the General Parallel File System (GPFS) and node-local NVMe storage, known as the Burst Buffer (BB).

Results and Discussion

The analysis of 23,389 ML-specific jobs revealed several critical trends in leadership-scale I/O.

The Burst Buffer Paradox and Domain Disparity

While Burst Buffers are designed for the small, bursty I/O characteristic of ML, they remain significantly underutilized. Only about 4.5% of ML jobs utilized the Burst Buffer, with most domains relying heavily on the standard GPFS.

Figure 1: Distribution of ML jobs across science domains, comparing usage of GPFS and Burst Buffer storage.

As shown in Figure 1, Biology accounts for the highest volume of ML jobs, yet Computer Science users demonstrate more effective utilization of the Burst Buffer. In domains like Physics and Materials Science, users often perform small I/O directly on the parallel file system, likely due to the perceived manual overhead of managing data movement to the Burst Buffer.

I/O Characteristics and Performance

ML workloads on Summit are overwhelmingly dominated by small I/O operations; nearly 99% of all read/write calls are smaller than 10MB. This pattern is challenging for traditional parallel file systems like GPFS, which are optimized for large sequential streams.

Figure 2: Density plots of Bytes Read vs. Bytes Write for ML jobs on GPFS (left) and Burst Buffer (right), categorized by science domain.

Figure 2 illustrates the I/O intensity across different domains. "Flagship" jobs (the largest runs) are almost exclusively read-intensive. Utilizing the Burst Buffer provided substantial performance gains, with a 4.95x improvement in read performance and a 3.48x improvement in write performance compared to GPFS.

Conclusion

The rapid growth of ML in HPC—with over 50% of the year's ML I/O activity occurring in the final four months—underscores the need for optimized storage solutions. The current underutilization of Burst Buffers suggests that manual data management is a major barrier. Future HPC systems should implement transparent I/O optimization layers that automatically detect ML workloads and migrate data to high-speed local storage, allowing scientists to focus on discovery rather than infrastructure management.

Summary updated

Abstract

High performance computing (HPC) is no longer solely limited to traditional workloads such as simulation and modeling. With the increase in the popularity of machine learning (ML) and deep learning (DL) technologies, we are observing that an increasing number of HPC users are incorporating ML methods into their workflow and scientific discovery processes, across a wide spectrum of science domains such as biology, earth science, and physics. This gives rise to a diverse set of I/O patterns than the traditional checkpoint/restart-based HPC I/O behavior. The details of the I/O characteristics of such ML I/O workloads have not been studied extensively for large-scale leadership HPC systems. This paper aims to fill that gap by providing an in-depth analysis to gain an understanding of the I/O behavior of ML I/O workloads using darshan - an I/O characterization tool designed for lightweight tracing and profiling. We study the darshan logs of more than 23, 000 HPC ML I/O jobs over a time period of one year running on Summit - the second-fastest supercomputer in the world. This paper provides a systematic I/O characterization of ML I/O jobs running on a leadership scale supercomputer to understand how the I/O behavior differs across science domains and the scale of workloads, and analyze the usage of parallel file system and burst buffer by ML I/O workloads.
BibTeX
@inproceedings{paul2021characterizing,
  author = {Arnab Kumar Paul and Ahmad Maroof Karimi and Feiyi Wang},
  title = {{Characterizing Machine Learning I/O Workloads on Leadership Scale HPC Systems}},
  booktitle = {MASCOTS},
  year = {2021},
  url = {https://openreview.net/forum?id=i43Vhei5As},
}
Figure