Abstract
High performance computing (HPC) is no longer solely limited to traditional workloads such as simulation and modeling. With the increase in the popularity of machine learning (ML) and deep learning (DL) technologies, an increasing number of HPC users are incorporating ML methods into their workflow and scientific discovery processes. This gives rise to a diverse set of I/O patterns compared to traditional checkpoint/restart-based HPC behavior. This paper provides an in-depth analysis of ML I/O behavior using Darshan logs from over 23,000 jobs on the Summit supercomputer over one year. We characterize how I/O behavior differs across science domains and analyze the usage of parallel file systems versus burst buffers.
Introduction
Traditionally, High-Performance Computing (HPC) was dominated by massive simulations that are "write-heavy," saving large checkpoints to stable storage. However, the rise of Machine Learning (ML) and Deep Learning (DL) has shifted the landscape. ML workloads are typically "read-intensive," involving millions of small files and random access patterns. Despite this shift, there is a lack of holistic understanding regarding ML workload behavior at the "leadership scale" (thousands of nodes). This study bridges that gap by analyzing Darshan profiling data from the Summit supercomputer to provide a roadmap for designing storage systems for next-generation AI-driven science.
Methodology
To extract ML-specific data from general HPC logs, a multi-step pipeline was developed:
- Keyword Annotation: A list of 42 keywords (e.g.,
pytorch,tensorflow,horovod) was used to scan Darshan logs for ML-related executables and files. - Data Integration: Darshan logs were merged with Summit’s scheduler logs to associate jobs with science domains and node counts.
- Classification: Jobs were categorized by scale (Small, Medium, Flagship) and I/O type (Read-Intensive, Write-Intensive, or Read-Write).
- Storage Comparison: Performance was compared between the General Parallel File System (GPFS) and node-local NVMe storage, known as the Burst Buffer (BB).
Results and Discussion
The analysis of 23,389 ML-specific jobs revealed several critical trends in leadership-scale I/O.
The Burst Buffer Paradox and Domain Disparity
While Burst Buffers are designed for the small, bursty I/O characteristic of ML, they remain significantly underutilized. Only about 4.5% of ML jobs utilized the Burst Buffer, with most domains relying heavily on the standard GPFS.
As shown in Figure 1, Biology accounts for the highest volume of ML jobs, yet Computer Science users demonstrate more effective utilization of the Burst Buffer. In domains like Physics and Materials Science, users often perform small I/O directly on the parallel file system, likely due to the perceived manual overhead of managing data movement to the Burst Buffer.
I/O Characteristics and Performance
ML workloads on Summit are overwhelmingly dominated by small I/O operations; nearly 99% of all read/write calls are smaller than 10MB. This pattern is challenging for traditional parallel file systems like GPFS, which are optimized for large sequential streams.
Figure 2 illustrates the I/O intensity across different domains. "Flagship" jobs (the largest runs) are almost exclusively read-intensive. Utilizing the Burst Buffer provided substantial performance gains, with a 4.95x improvement in read performance and a 3.48x improvement in write performance compared to GPFS.
Conclusion
The rapid growth of ML in HPC—with over 50% of the year's ML I/O activity occurring in the final four months—underscores the need for optimized storage solutions. The current underutilization of Burst Buffers suggests that manual data management is a major barrier. Future HPC systems should implement transparent I/O optimization layers that automatically detect ML workloads and migrate data to high-speed local storage, allowing scientists to focus on discovery rather than infrastructure management.
Summary updated